[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:33:25 dist-test-slave-nf4n krb5kdc[29098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:33:25 dist-test-slave-nf4n krb5kdc[29098](info): set up 2 sockets
Jun 19 06:33:25 dist-test-slave-nf4n krb5kdc[29098](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 06:33:26.666556 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:33:26 dist-test-slave-nf4n krb5kdc[29098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:33:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:33:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:33:26.849992 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:33703
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:33703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:27.176330 29111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:27.176687 29111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:27.176859 29111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:27.184198 29111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:27.184365 29111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:27.184489 29111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:27.184598 29111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:27.184681 29111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:27.194876 29111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:33703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:33703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:27.196300 29111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:27.197803 29111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:27.210326 29119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:27.210398 29117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:27.210354 29116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:27.211196 29111 server_base.cc:1034] running on GCE node
I20260619 06:33:27.212127 29111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:27.213783 29111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:27.215047 29111 hybrid_clock.cc:648] HybridClock initialized: now 1781850807214987 us; error 48 us; skew 500 ppm
I20260619 06:33:27.215461 29111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:27 dist-test-slave-nf4n krb5kdc[29098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:27.218677 29111 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:33:27.220139 29111 webserver.cc:460] Webserver started at http://127.28.104.254:40957/ using document root <none> and password file <none>
I20260619 06:33:27.220818 29111 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:27.220925 29111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:27.221246 29111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:27.224097 29111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "261a184c8f3a46278ecd43029f2121be"
format_stamp: "Formatted at 2026-06-19 06:33:27 on dist-test-slave-nf4n"
server_key: "442bdf874cbe151e4d2ccea57e18f074"
server_key_iv: "349851f2faa1d77a9a2fc825ccd6f77a"
server_key_version: "encryptionkey@0"
I20260619 06:33:27.224915 29111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "261a184c8f3a46278ecd43029f2121be"
format_stamp: "Formatted at 2026-06-19 06:33:27 on dist-test-slave-nf4n"
server_key: "442bdf874cbe151e4d2ccea57e18f074"
server_key_iv: "349851f2faa1d77a9a2fc825ccd6f77a"
server_key_version: "encryptionkey@0"
I20260619 06:33:27.230165 29111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:33:27.233546 29126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:27.234951 29111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:33:27.235113 29111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "261a184c8f3a46278ecd43029f2121be"
format_stamp: "Formatted at 2026-06-19 06:33:27 on dist-test-slave-nf4n"
server_key: "442bdf874cbe151e4d2ccea57e18f074"
server_key_iv: "349851f2faa1d77a9a2fc825ccd6f77a"
server_key_version: "encryptionkey@0"
I20260619 06:33:27.235311 29111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:27.252867 29111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:27.257153 29111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:27.257443 29111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:27.276190 29111 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:33703
I20260619 06:33:27.276221 29177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:33703 every 8 connection(s)
I20260619 06:33:27.277741 29111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:33:27.281663 29178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:27.286077 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 29111
I20260619 06:33:27.286391 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:33:27.288233 29091 external_mini_cluster.cc:1428] Setting key 6e01f5ad66943f346706e48f5432da5e
I20260619 06:33:27.292106 29178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be: Bootstrap starting.
I20260619 06:33:27.295475 29178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:27.296533 29178 log.cc:824] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:27.299419 29178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be: No bootstrap required, opened a new log
I20260619 06:33:27.309362 29178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261a184c8f3a46278ecd43029f2121be" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33703 } }
I20260619 06:33:27.309876 29178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:27.310037 29178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261a184c8f3a46278ecd43029f2121be, State: Initialized, Role: FOLLOWER
I20260619 06:33:27.310724 29178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261a184c8f3a46278ecd43029f2121be" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33703 } }
I20260619 06:33:27.311009 29178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:33:27.311154 29178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:33:27.311332 29178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:27.314293 29178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261a184c8f3a46278ecd43029f2121be" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33703 } }
I20260619 06:33:27.314805 29178 leader_election.cc:302] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261a184c8f3a46278ecd43029f2121be; no voters:
I20260619 06:33:27.315486 29178 leader_election.cc:288] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:33:27.315629 29183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:27.317656 29178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:27.319675 29183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [term 1 LEADER]: Becoming Leader. State: Replica: 261a184c8f3a46278ecd43029f2121be, State: Running, Role: LEADER
Jun 19 06:33:27 dist-test-slave-nf4n krb5kdc[29098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:27.320348 29183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261a184c8f3a46278ecd43029f2121be" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33703 } }
I20260619 06:33:27.323421 29185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261a184c8f3a46278ecd43029f2121be. Latest consensus state: current_term: 1 leader_uuid: "261a184c8f3a46278ecd43029f2121be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261a184c8f3a46278ecd43029f2121be" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33703 } } }
I20260619 06:33:27.323858 29185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:27.324352 29184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261a184c8f3a46278ecd43029f2121be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261a184c8f3a46278ecd43029f2121be" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33703 } } }
I20260619 06:33:27.324605 29184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:27.327692 29193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:33:27.334350 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:27.292259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43772 (local address 127.28.104.254:33703)
0619 06:33:27.293212 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:27.293227 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:27.294616 (+ 1389us) server_negotiation.cc:407] Connection header received
0619 06:33:27.304557 (+ 9941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:27.304591 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:27.305253 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:27.305784 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:27.308610 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:27.310631 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:27.312521 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:27.313645 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:27.313828 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:27.322071 (+ 8243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:27.322088 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:27.322121 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:27.322184 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:27.327037 (+ 4853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:27.328300 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:27.328314 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:27.328326 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:27.328614 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:27.329261 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:27.329269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:27.329274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:27.330383 (+ 1109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:27.330853 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:27.331921 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0619 06:33:27.332502 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":441,"threads_started":1}
I20260619 06:33:27.338078 29193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:33:27.349062 29193 catalog_manager.cc:1347] Generated new cluster ID: 00437dc0be9e4621990d10ce0a4fcc8e
I20260619 06:33:27.349329 29193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:33:27.362994 29193 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:33:27.364938 29193 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:33:27.374259 29193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261a184c8f3a46278ecd43029f2121be: Generated new TSK 0
I20260619 06:33:27.375344 29193 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:33:27.420926 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33703
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:27.749194 29204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:27.749476 29204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:27.749670 29204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:27.757726 29204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:27.757861 29204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:27.758102 29204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:33:27.768492 29204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:27.770007 29204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:27.771509 29204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:27.785634 29209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:27.785665 29210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:27.785758 29212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:27.786429 29204 server_base.cc:1034] running on GCE node
I20260619 06:33:27.787071 29204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:27.788100 29204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:27.789361 29204 hybrid_clock.cc:648] HybridClock initialized: now 1781850807789315 us; error 49 us; skew 500 ppm
I20260619 06:33:27.789769 29204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:27 dist-test-slave-nf4n krb5kdc[29098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:27.793134 29204 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:33:27.794695 29204 webserver.cc:460] Webserver started at http://127.28.104.193:43897/ using document root <none> and password file <none>
I20260619 06:33:27.795393 29204 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:27.795536 29204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:27.795890 29204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:27.799093 29204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "012ceba3cbb74689a66b7fd9f9c69a31"
format_stamp: "Formatted at 2026-06-19 06:33:27 on dist-test-slave-nf4n"
server_key: "dfc25482d0649b38413510631d36a511"
server_key_iv: "910162c5e1375da40a9689566e9bf9c0"
server_key_version: "encryptionkey@0"
I20260619 06:33:27.799866 29204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "012ceba3cbb74689a66b7fd9f9c69a31"
format_stamp: "Formatted at 2026-06-19 06:33:27 on dist-test-slave-nf4n"
server_key: "dfc25482d0649b38413510631d36a511"
server_key_iv: "910162c5e1375da40a9689566e9bf9c0"
server_key_version: "encryptionkey@0"
I20260619 06:33:27.805135 29204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:33:27.808655 29219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:27.810093 29204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:27.810271 29204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "012ceba3cbb74689a66b7fd9f9c69a31"
format_stamp: "Formatted at 2026-06-19 06:33:27 on dist-test-slave-nf4n"
server_key: "dfc25482d0649b38413510631d36a511"
server_key_iv: "910162c5e1375da40a9689566e9bf9c0"
server_key_version: "encryptionkey@0"
I20260619 06:33:27.810508 29204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:27.830482 29204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:27.835665 29204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:27.835984 29204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:27.837119 29204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:27.838467 29204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:27.838569 29204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:27.838696 29204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:27.838776 29204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:27.872206 29204 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:37041
I20260619 06:33:27.872253 29331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:37041 every 8 connection(s)
I20260619 06:33:27.873714 29204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:33:27.879495 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 29204
I20260619 06:33:27.879733 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:33:27.880520 29091 external_mini_cluster.cc:1428] Setting key f5e87ea8fa4eb1126b1f3a49371c8f3b
Jun 19 06:33:27 dist-test-slave-nf4n krb5kdc[29098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:27.896486 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:27.876283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:55481 (local address 127.28.104.254:33703)
0619 06:33:27.876731 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:27.876737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:27.877880 (+ 1143us) server_negotiation.cc:407] Connection header received
0619 06:33:27.880102 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:27.880109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:27.880206 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:27.880322 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:27.882169 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:27.883190 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:27.884775 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:27.885512 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:27.885654 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:27.889658 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:27.889668 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:27.889676 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:27.889721 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:27.893158 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:27.893980 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:27.893988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:27.893991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:27.894135 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:27.894541 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:27.894550 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:27.894555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:27.894961 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:27.895134 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:27.895822 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 06:33:27.896065 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":210,"threads_started":1}
I20260619 06:33:27.897272 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:27.876976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:33703 (local address 127.28.104.193:55481)
0619 06:33:27.877723 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:27.877761 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:27.879661 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:27.880617 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:27.880629 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:27.881256 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:27.881996 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:27.882022 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:27.883367 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:27.883371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:27.884622 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:27.884634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:27.885746 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:27.885751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:27.885969 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:27.886692 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:27.886732 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:27.889439 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:27.893286 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:27.893292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:27.893304 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:27.893840 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:27.894243 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:27.894246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:27.894248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:27.894412 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:27.895096 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:27.895103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:27.895421 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 06:33:27.895652 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 06:33:27.896027 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":246,"threads_started":1}
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
I20260619 06:33:27.899015 29332 heartbeater.cc:347] Connected to a master server at 127.28.104.254:33703
I20260619 06:33:27.899335 29332 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:27.900166 29332 heartbeater.cc:511] Master 127.28.104.254:33703 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
I20260619 06:33:27.902515 29143 ts_manager.cc:194] Registered new tserver with Master: 012ceba3cbb74689a66b7fd9f9c69a31 (127.28.104.193:37041)
I20260619 06:33:27.903990 29143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:55481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:33:27.922086 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33703
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:28.249362 29338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:28.249629 29338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:28.249723 29338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:28.256845 29338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:28.256960 29338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:28.257174 29338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:33:28.266974 29338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:28.268288 29338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:28.269675 29338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:28.284150 29343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:28.284134 29344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:28.284837 29346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:28.285580 29338 server_base.cc:1034] running on GCE node
I20260619 06:33:28.286248 29338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:28.287369 29338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:28.288596 29338 hybrid_clock.cc:648] HybridClock initialized: now 1781850808288547 us; error 37 us; skew 500 ppm
I20260619 06:33:28.288946 29338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:28 dist-test-slave-nf4n krb5kdc[29098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:28.292673 29338 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:33:28.294385 29338 webserver.cc:460] Webserver started at http://127.28.104.194:42001/ using document root <none> and password file <none>
I20260619 06:33:28.295091 29338 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:28.295233 29338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:28.295542 29338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:28.298766 29338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "4fdab647c2544d9aa0b7be9b0de8b1b2"
format_stamp: "Formatted at 2026-06-19 06:33:28 on dist-test-slave-nf4n"
server_key: "65d8018928d1a4d352baaff6f7a6013e"
server_key_iv: "0ac0ef5d55172bd39c3e27e37dab896d"
server_key_version: "encryptionkey@0"
I20260619 06:33:28.299716 29338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "4fdab647c2544d9aa0b7be9b0de8b1b2"
format_stamp: "Formatted at 2026-06-19 06:33:28 on dist-test-slave-nf4n"
server_key: "65d8018928d1a4d352baaff6f7a6013e"
server_key_iv: "0ac0ef5d55172bd39c3e27e37dab896d"
server_key_version: "encryptionkey@0"
I20260619 06:33:28.305240 29338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:33:28.308938 29353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:28.310379 29338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:33:28.310560 29338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "4fdab647c2544d9aa0b7be9b0de8b1b2"
format_stamp: "Formatted at 2026-06-19 06:33:28 on dist-test-slave-nf4n"
server_key: "65d8018928d1a4d352baaff6f7a6013e"
server_key_iv: "0ac0ef5d55172bd39c3e27e37dab896d"
server_key_version: "encryptionkey@0"
I20260619 06:33:28.310763 29338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:28.336032 29338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:28.340659 29338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:28.340970 29338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:28.342051 29338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:28.343605 29338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:28.343715 29338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:28.343870 29338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:28.343943 29338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:28.378058 29338 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:46813
I20260619 06:33:28.378149 29465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:46813 every 8 connection(s)
I20260619 06:33:28.379438 29338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:33:28.389003 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 29338
I20260619 06:33:28.389295 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:33:28.389750 29091 external_mini_cluster.cc:1428] Setting key 4ff22ba302fb8ef9789085dcdd8c2b14
Jun 19 06:33:28 dist-test-slave-nf4n krb5kdc[29098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:28.401512 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:28.382283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:35603 (local address 127.28.104.254:33703)
0619 06:33:28.382531 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:28.382538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:28.383780 (+ 1242us) server_negotiation.cc:407] Connection header received
0619 06:33:28.385652 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:28.385657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:28.385732 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:28.385828 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:28.387317 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:28.388133 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:28.389885 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:28.390351 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:28.390491 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:28.394638 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:28.394655 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:28.394663 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:28.394706 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:28.398051 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:28.399083 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:28.399087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:28.399089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:28.399215 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:28.399732 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:28.399735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:28.399737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:28.399977 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:28.400081 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:28.400999 (+ 918us) server_negotiation.cc:299] Negotiation successful
0619 06:33:28.401270 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 06:33:28.403256 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:28.382864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:33703 (local address 127.28.104.194:35603)
0619 06:33:28.383587 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:28.383635 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:28.385379 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:28.386040 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:28.386049 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:28.386554 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:28.387144 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:28.387161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:28.388274 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:28.388278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:28.389741 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:28.389752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:28.390619 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:28.390627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:28.390938 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:28.391755 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:28.391779 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:28.394400 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:28.398201 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:28.398214 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:28.398235 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:28.398954 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:28.399326 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:28.399334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:28.399342 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:28.399617 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:28.400121 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:28.400133 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:28.400600 (+ 467us) client_negotiation.cc:769] Sending connection context
0619 06:33:28.400932 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 06:33:28.401566 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":278,"threads_started":1}
I20260619 06:33:28.404894 29466 heartbeater.cc:347] Connected to a master server at 127.28.104.254:33703
I20260619 06:33:28.405272 29466 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:28.405997 29466 heartbeater.cc:511] Master 127.28.104.254:33703 requested a full tablet report, sending...
I20260619 06:33:28.407505 29143 ts_manager.cc:194] Registered new tserver with Master: 4fdab647c2544d9aa0b7be9b0de8b1b2 (127.28.104.194:46813)
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
I20260619 06:33:28.408340 29143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:35603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:33:28.431800 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33703
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:28.752328 29471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:28.752686 29471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:28.752871 29471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:28.760213 29471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:28.760370 29471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:28.760561 29471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:33:28.770623 29471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:28.772042 29471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:28.773417 29471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:28.790180 29479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:28.790326 29476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:28.790302 29477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:28.791433 29471 server_base.cc:1034] running on GCE node
I20260619 06:33:28.792032 29471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:28.792996 29471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:28.794278 29471 hybrid_clock.cc:648] HybridClock initialized: now 1781850808794227 us; error 49 us; skew 500 ppm
I20260619 06:33:28.794606 29471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:28 dist-test-slave-nf4n krb5kdc[29098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:28.797792 29471 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:33:28.799261 29471 webserver.cc:460] Webserver started at http://127.28.104.195:34761/ using document root <none> and password file <none>
I20260619 06:33:28.799949 29471 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:28.800089 29471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:28.800398 29471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:28.804306 29471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "616840cbdd4b46f9ab9dff240f738fe2"
format_stamp: "Formatted at 2026-06-19 06:33:28 on dist-test-slave-nf4n"
server_key: "6fa1d393072f4877ad4c0dc68f421a55"
server_key_iv: "7e9aca1da9cd4f1f46df9d8d8d890a9d"
server_key_version: "encryptionkey@0"
I20260619 06:33:28.805138 29471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "616840cbdd4b46f9ab9dff240f738fe2"
format_stamp: "Formatted at 2026-06-19 06:33:28 on dist-test-slave-nf4n"
server_key: "6fa1d393072f4877ad4c0dc68f421a55"
server_key_iv: "7e9aca1da9cd4f1f46df9d8d8d890a9d"
server_key_version: "encryptionkey@0"
I20260619 06:33:28.810355 29471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 06:33:28.814429 29486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:28.816447 29471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:33:28.816639 29471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "616840cbdd4b46f9ab9dff240f738fe2"
format_stamp: "Formatted at 2026-06-19 06:33:28 on dist-test-slave-nf4n"
server_key: "6fa1d393072f4877ad4c0dc68f421a55"
server_key_iv: "7e9aca1da9cd4f1f46df9d8d8d890a9d"
server_key_version: "encryptionkey@0"
I20260619 06:33:28.816866 29471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:28.849437 29471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:28.854187 29471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:28.854502 29471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:28.855530 29471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:28.857025 29471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:28.857138 29471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 06:33:28.857278 29471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:28.857344 29471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:28.892920 29471 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:38015
I20260619 06:33:28.892964 29598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:38015 every 8 connection(s)
I20260619 06:33:28.894255 29471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:33:28.898332 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 29471
I20260619 06:33:28.898563 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:33:28.899132 29091 external_mini_cluster.cc:1428] Setting key 458bf9b92d05625d876627eca568307f
I20260619 06:33:28.907658 29332 heartbeater.cc:503] Master 127.28.104.254:33703 was elected leader, sending a full tablet report...
Jun 19 06:33:28 dist-test-slave-nf4n krb5kdc[29098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:28.916494 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:28.896824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:60809 (local address 127.28.104.254:33703)
0619 06:33:28.897074 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:28.897081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:28.898480 (+ 1399us) server_negotiation.cc:407] Connection header received
0619 06:33:28.901150 (+ 2670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:28.901155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:28.901220 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:28.901303 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:28.903630 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:28.904384 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:28.906050 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:28.906442 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:28.906546 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:28.910293 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:28.910298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:28.910301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:28.910330 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:28.912968 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:28.913913 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:28.913920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:28.913924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:28.914178 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:28.914740 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:28.914747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:28.914751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:28.915147 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:28.915284 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:28.916096 (+ 812us) server_negotiation.cc:299] Negotiation successful
0619 06:33:28.916248 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 06:33:28.917582 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:28.897466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:33703 (local address 127.28.104.195:60809)
0619 06:33:28.898301 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:28.898357 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:28.900761 (+ 2404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:28.901599 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:28.901612 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:28.902223 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:28.903474 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:28.903494 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:28.904537 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:28.904541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:28.905925 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:28.905937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:28.906631 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:28.906635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:28.906851 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:28.907526 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:28.907549 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:28.910120 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:28.913151 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:28.913158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:28.913169 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:28.913760 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:28.914361 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:28.914368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:28.914372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:28.914622 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:28.915282 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:28.915290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:28.915770 (+ 480us) client_negotiation.cc:769] Sending connection context
0619 06:33:28.916041 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 06:33:28.916427 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":268,"threads_started":1}
I20260619 06:33:28.918946 29599 heartbeater.cc:347] Connected to a master server at 127.28.104.254:33703
I20260619 06:33:28.919282 29599 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:28.919982 29599 heartbeater.cc:511] Master 127.28.104.254:33703 requested a full tablet report, sending...
I20260619 06:33:28.921702 29143 ts_manager.cc:194] Registered new tserver with Master: 616840cbdd4b46f9ab9dff240f738fe2 (127.28.104.195:38015)
I20260619 06:33:28.922595 29143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:60809
I20260619 06:33:28.927829 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:33:28 dist-test-slave-nf4n krb5kdc[29098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:33:29 dist-test-slave-nf4n krb5kdc[29098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:33:29.030720 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:29.013661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.28.104.193:37041)
0619 06:33:29.015383 (+ 1722us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:29.015391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:29.015418 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:33:29.015518 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:29.015525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:29.015759 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:29.015890 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:29.017959 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:29.019094 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:29.020326 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:29.020978 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:29.021172 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:29.025179 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:29.025190 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:29.025207 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:29.025240 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:29.028231 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:29.028833 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:29.028840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:29.028842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:29.028962 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:29.029328 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:29.029332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:29.029334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:29.029674 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:29.029900 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:29.030100 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 06:33:29.030284 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1539,"thread_start_us":1369,"threads_started":1}
W20260619 06:33:29.031970 29247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44888
Jun 19 06:33:29 dist-test-slave-nf4n krb5kdc[29098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:33:29 dist-test-slave-nf4n krb5kdc[29098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:33:29.133258 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:29.118228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.28.104.193:37041)
0619 06:33:29.118457 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:29.118467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:29.119021 (+ 554us) server_negotiation.cc:407] Connection header received
0619 06:33:29.119185 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:29.119193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:29.119327 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:29.119451 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:29.121118 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:29.121903 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:29.123209 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:29.123618 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:29.123714 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:29.127840 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:29.127845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:29.127848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:29.127880 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:29.130562 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:29.131235 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:29.131247 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:29.131251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:29.131422 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:29.131811 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:29.131816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:29.131819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:29.132086 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:29.132328 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:29.132645 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 06:33:29.132891 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:33:29.134850 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 29204
I20260619 06:33:29.148762 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:29.135962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.28.104.193:37041)
0619 06:33:29.136182 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:29.136190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:29.136428 (+ 238us) server_negotiation.cc:407] Connection header received
0619 06:33:29.136662 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:29.136666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:29.136727 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:29.136837 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:29.138169 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:29.138980 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:29.140347 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:29.140789 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:29.140951 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:29.143427 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:29.143433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:29.143436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:29.143469 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:29.146198 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:29.146879 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:29.146887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:29.146890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:29.147020 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:29.147385 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:29.147393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:29.147397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:29.147657 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:29.147927 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:29.148215 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 06:33:29.148419 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:33:29.149551 29327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:29.235668 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 29204
I20260619 06:33:29.253355 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 29338
Jun 19 06:33:29 dist-test-slave-nf4n krb5kdc[29098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:33:29.269683 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:29.254265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.28.104.194:46813)
0619 06:33:29.254902 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:29.254910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:29.254937 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:33:29.255001 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:29.255008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:29.255221 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:29.255362 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:29.256510 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:29.257663 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:29.258887 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:29.259551 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:29.259671 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:29.263948 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:29.263961 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:29.263977 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:29.264019 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:29.266829 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:29.267437 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:29.267444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:29.267446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:29.267557 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:29.267963 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:29.267967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:29.267969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:29.268347 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:29.268665 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:29.268933 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 06:33:29.269228 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":315,"threads_started":1}
I20260619 06:33:29.271122 29461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:29.361392 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 29338
I20260619 06:33:29.384224 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 29471
Jun 19 06:33:29 dist-test-slave-nf4n krb5kdc[29098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:33:29.401961 29615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:29.385397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.28.104.195:38015)
0619 06:33:29.385889 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:29.385896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:29.385927 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:33:29.385991 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:29.385997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:29.386217 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:29.386354 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:29.387737 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:29.388700 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:29.390150 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:29.390700 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:29.390797 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:29.395683 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:29.395697 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:29.395712 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:29.395779 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:29.399081 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:29.399766 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:29.399774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:29.399776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:29.399895 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:29.400326 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:29.400331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:29.400334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:29.400720 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:29.401083 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:29.401326 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 06:33:29.401595 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":236,"threads_started":1}
I20260619 06:33:29.403240 29594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:29.488215 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 29471
I20260619 06:33:29.505823 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 29111
I20260619 06:33:29.507154 29173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:29.613432 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 29111
2026-06-19T06:33:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4060 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:33:31 dist-test-slave-nf4n krb5kdc[29620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:33:31 dist-test-slave-nf4n krb5kdc[29620](info): set up 2 sockets
Jun 19 06:33:31 dist-test-slave-nf4n krb5kdc[29620](info): commencing operation
krb5kdc: starting...
W20260619 06:33:32.576679 29091 mini_kdc.cc:121] Time spent starting KDC: real 2.919s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 06:33:32.897953 29091 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.321s user 0.003s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 06:33:33.137488 29091 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.239s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 06:33:33.409626 29091 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.272s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/test-user.keytab.
W20260619 06:33:34.246187 29091 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.836s user 0.000s sys 0.004s
Jun 19 06:33:34 dist-test-slave-nf4n krb5kdc[29620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:33:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:33:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
W20260619 06:33:34.447458 29091 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.104.254: real 0.153s user 0.002s sys 0.006s
I20260619 06:33:34.448449 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:35365
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:35365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:34.768810 29633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:34.769268 29633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:34.769382 29633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:34.776964 29633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:34.777154 29633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:34.777281 29633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:34.777401 29633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:34.777472 29633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:34.789011 29633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:35365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:35365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:34.790567 29633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:34.792037 29633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:34.805079 29638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:34.805088 29639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:34.805325 29641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:34.806299 29633 server_base.cc:1034] running on GCE node
I20260619 06:33:34.807180 29633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:34.808681 29633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:34.809894 29633 hybrid_clock.cc:648] HybridClock initialized: now 1781850814809847 us; error 40 us; skew 500 ppm
I20260619 06:33:34.810290 29633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:34 dist-test-slave-nf4n krb5kdc[29620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850814, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:34.813571 29633 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:33:34.815145 29633 webserver.cc:460] Webserver started at http://127.28.104.254:45371/ using document root <none> and password file <none>
I20260619 06:33:34.815889 29633 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:34.816076 29633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:34.816555 29633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:34.819850 29633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "2258f1be57814a04be4ba5f433808f2f"
format_stamp: "Formatted at 2026-06-19 06:33:34 on dist-test-slave-nf4n"
server_key: "16027a70333621a80401a5db77ef9616"
server_key_iv: "d9cba31ab49eb1bd782889112e4afe7d"
server_key_version: "encryptionkey@0"
I20260619 06:33:34.820672 29633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "2258f1be57814a04be4ba5f433808f2f"
format_stamp: "Formatted at 2026-06-19 06:33:34 on dist-test-slave-nf4n"
server_key: "16027a70333621a80401a5db77ef9616"
server_key_iv: "d9cba31ab49eb1bd782889112e4afe7d"
server_key_version: "encryptionkey@0"
I20260619 06:33:34.826366 29633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 06:33:34.830219 29648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:34.831633 29633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260619 06:33:34.831835 29633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "2258f1be57814a04be4ba5f433808f2f"
format_stamp: "Formatted at 2026-06-19 06:33:34 on dist-test-slave-nf4n"
server_key: "16027a70333621a80401a5db77ef9616"
server_key_iv: "d9cba31ab49eb1bd782889112e4afe7d"
server_key_version: "encryptionkey@0"
I20260619 06:33:34.832063 29633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:34.854446 29633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:34.858897 29633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:34.859229 29633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:34.877825 29633 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:35365
I20260619 06:33:34.877844 29699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:35365 every 8 connection(s)
I20260619 06:33:34.879300 29633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:33:34.883077 29700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:34.885428 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 29633
I20260619 06:33:34.885643 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:33:34.886439 29091 external_mini_cluster.cc:1428] Setting key 3c28505a191c0b822e2b8ff15dc5bc3c
I20260619 06:33:34.895093 29700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f: Bootstrap starting.
I20260619 06:33:34.898187 29700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:34.899312 29700 log.cc:824] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:33:34 dist-test-slave-nf4n krb5kdc[29620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:34.902719 29700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f: No bootstrap required, opened a new log
I20260619 06:33:34.910953 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:34.888728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.28.104.254:35365)
0619 06:33:34.889562 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:34.889583 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:34.889632 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:33:34.891402 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:34.891426 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:34.891852 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:34.892296 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:34.894044 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:34.895486 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:34.896814 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:34.897656 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:34.897863 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:34.903036 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:34.903046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:34.903061 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:34.903109 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:34.906521 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:34.907359 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:34.907365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:34.907371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:34.907549 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:34.907947 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:34.907950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:34.907952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:34.908476 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:34.908715 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:34.909355 (+ 640us) server_negotiation.cc:299] Negotiation successful
0619 06:33:34.909809 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":367,"threads_started":1}
I20260619 06:33:34.911815 29700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258f1be57814a04be4ba5f433808f2f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35365 } }
I20260619 06:33:34.912216 29700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:34.912343 29700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2258f1be57814a04be4ba5f433808f2f, State: Initialized, Role: FOLLOWER
I20260619 06:33:34.913090 29700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258f1be57814a04be4ba5f433808f2f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35365 } }
I20260619 06:33:34.913384 29700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:33:34.913516 29700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:33:34.913690 29700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:34.915964 29700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258f1be57814a04be4ba5f433808f2f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35365 } }
I20260619 06:33:34.916397 29700 leader_election.cc:302] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2258f1be57814a04be4ba5f433808f2f; no voters:
I20260619 06:33:34.916998 29700 leader_election.cc:288] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:33:34.917155 29705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:34.917819 29705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [term 1 LEADER]: Becoming Leader. State: Replica: 2258f1be57814a04be4ba5f433808f2f, State: Running, Role: LEADER
I20260619 06:33:34.918578 29705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258f1be57814a04be4ba5f433808f2f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35365 } }
I20260619 06:33:34.919744 29700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:34.921770 29706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2258f1be57814a04be4ba5f433808f2f. Latest consensus state: current_term: 1 leader_uuid: "2258f1be57814a04be4ba5f433808f2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258f1be57814a04be4ba5f433808f2f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35365 } } }
I20260619 06:33:34.921620 29707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2258f1be57814a04be4ba5f433808f2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258f1be57814a04be4ba5f433808f2f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35365 } } }
I20260619 06:33:34.923266 29707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:34.923847 29706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:34.925257 29714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:33:34.930712 29714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:33:34.940240 29714 catalog_manager.cc:1347] Generated new cluster ID: 05235db9c843444faa517c0a41d543fb
I20260619 06:33:34.940394 29714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:33:34.975893 29714 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:33:34.977499 29714 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:33:34.989674 29714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2258f1be57814a04be4ba5f433808f2f: Generated new TSK 0
I20260619 06:33:34.990573 29714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:33:35.033125 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35365
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:35.365906 29726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:35.366282 29726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:35.366463 29726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:35.374006 29726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:35.374169 29726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:35.374365 29726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:33:35.384305 29726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:35.385838 29726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:35.387245 29726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:35.401600 29731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:35.401661 29732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:35.401638 29734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:35.402788 29726 server_base.cc:1034] running on GCE node
I20260619 06:33:35.403460 29726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:35.404403 29726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:35.405602 29726 hybrid_clock.cc:648] HybridClock initialized: now 1781850815405550 us; error 38 us; skew 500 ppm
I20260619 06:33:35.405932 29726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:35 dist-test-slave-nf4n krb5kdc[29620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850815, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:35.414691 29726 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:33:35.416303 29726 webserver.cc:460] Webserver started at http://127.28.104.193:38319/ using document root <none> and password file <none>
I20260619 06:33:35.417084 29726 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:35.417240 29726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:35.417560 29726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:35.420614 29726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "b202f5428a124f1489ed3b723d663809"
format_stamp: "Formatted at 2026-06-19 06:33:35 on dist-test-slave-nf4n"
server_key: "f2b94e04c63d827b1f1f3adc4b70eb74"
server_key_iv: "b7de6759182ca9c829d8950653ff19c5"
server_key_version: "encryptionkey@0"
I20260619 06:33:35.421478 29726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "b202f5428a124f1489ed3b723d663809"
format_stamp: "Formatted at 2026-06-19 06:33:35 on dist-test-slave-nf4n"
server_key: "f2b94e04c63d827b1f1f3adc4b70eb74"
server_key_iv: "b7de6759182ca9c829d8950653ff19c5"
server_key_version: "encryptionkey@0"
I20260619 06:33:35.426872 29726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:33:35.430372 29741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:35.431816 29726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:33:35.432093 29726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "b202f5428a124f1489ed3b723d663809"
format_stamp: "Formatted at 2026-06-19 06:33:35 on dist-test-slave-nf4n"
server_key: "f2b94e04c63d827b1f1f3adc4b70eb74"
server_key_iv: "b7de6759182ca9c829d8950653ff19c5"
server_key_version: "encryptionkey@0"
I20260619 06:33:35.432314 29726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:35.460537 29726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:35.466387 29726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:35.466744 29726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:35.467927 29726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:35.469619 29726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:35.469707 29726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:35.469830 29726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:35.469888 29726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:35.505405 29726 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:38023
I20260619 06:33:35.505512 29853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:38023 every 8 connection(s)
I20260619 06:33:35.506731 29726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:33:35.512444 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 29726
I20260619 06:33:35.512682 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:33:35.513470 29091 external_mini_cluster.cc:1428] Setting key d893642eec17a851353510f6615ac15e
Jun 19 06:33:35 dist-test-slave-nf4n krb5kdc[29620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850815, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:35.528692 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:35.509385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:47689 (local address 127.28.104.254:35365)
0619 06:33:35.509812 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:35.509817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:35.510869 (+ 1052us) server_negotiation.cc:407] Connection header received
0619 06:33:35.512760 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:35.512770 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:35.512874 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:35.513040 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:35.514679 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:35.515703 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:35.518224 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:35.518629 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:35.518724 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:35.522804 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:35.522809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:35.522812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:35.522844 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:35.525615 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:35.526363 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:35.526370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:35.526379 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:35.526501 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:35.526939 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:35.526944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:35.526947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:35.527215 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:35.527366 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:35.528088 (+ 722us) server_negotiation.cc:299] Negotiation successful
0619 06:33:35.528312 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":231,"threads_started":1}
I20260619 06:33:35.529770 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:35.510025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35365 (local address 127.28.104.193:47689)
0619 06:33:35.510704 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:35.510747 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:35.512432 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:35.513257 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:35.513265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:35.513837 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:35.514517 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:35.514538 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:35.515867 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:35.515873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:35.518079 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:35.518097 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:35.518859 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:35.518868 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:35.519185 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:35.519940 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:35.519980 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:35.522574 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:35.525769 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:35.525778 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:35.525795 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:35.526232 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:35.526613 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:35.526618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:35.526622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:35.526818 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:35.527354 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:35.527361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:35.527713 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 06:33:35.527994 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 06:33:35.528543 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":258,"threads_started":1}
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
I20260619 06:33:35.531436 29854 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35365
I20260619 06:33:35.531745 29854 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:35.532573 29854 heartbeater.cc:511] Master 127.28.104.254:35365 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
I20260619 06:33:35.534937 29665 ts_manager.cc:194] Registered new tserver with Master: b202f5428a124f1489ed3b723d663809 (127.28.104.193:38023)
I20260619 06:33:35.536643 29665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:47689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:33:35.555896 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35365
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:35.874996 29860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:35.875284 29860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:35.875378 29860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:35.882678 29860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:35.882781 29860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:35.882961 29860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:33:35.892975 29860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:35.894279 29860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:35.895629 29860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:35.908852 29865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:35.908787 29868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:35.908744 29866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:35.909934 29860 server_base.cc:1034] running on GCE node
I20260619 06:33:35.910545 29860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:35.911499 29860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:35.912735 29860 hybrid_clock.cc:648] HybridClock initialized: now 1781850815912689 us; error 41 us; skew 500 ppm
I20260619 06:33:35.913129 29860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:35 dist-test-slave-nf4n krb5kdc[29620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850815, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:35.920173 29860 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:33:35.922042 29860 webserver.cc:460] Webserver started at http://127.28.104.194:39243/ using document root <none> and password file <none>
I20260619 06:33:35.922691 29860 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:35.922788 29860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:35.923043 29860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:35.926177 29860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "5c85e161085b497b9127b660f2433c83"
format_stamp: "Formatted at 2026-06-19 06:33:35 on dist-test-slave-nf4n"
server_key: "3d632580f015cf280cde3ab59c5ecd39"
server_key_iv: "dd4e280c41a4a773fd6a7448b1075be8"
server_key_version: "encryptionkey@0"
I20260619 06:33:35.926929 29860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "5c85e161085b497b9127b660f2433c83"
format_stamp: "Formatted at 2026-06-19 06:33:35 on dist-test-slave-nf4n"
server_key: "3d632580f015cf280cde3ab59c5ecd39"
server_key_iv: "dd4e280c41a4a773fd6a7448b1075be8"
server_key_version: "encryptionkey@0"
I20260619 06:33:35.932226 29860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:33:35.935824 29875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:35.937435 29860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:33:35.937618 29860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "5c85e161085b497b9127b660f2433c83"
format_stamp: "Formatted at 2026-06-19 06:33:35 on dist-test-slave-nf4n"
server_key: "3d632580f015cf280cde3ab59c5ecd39"
server_key_iv: "dd4e280c41a4a773fd6a7448b1075be8"
server_key_version: "encryptionkey@0"
I20260619 06:33:35.937835 29860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:35.966284 29860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:35.970707 29860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:35.971047 29860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:35.972093 29860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:35.973659 29860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:35.973762 29860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:35.973917 29860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:35.973989 29860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:36.007524 29860 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:36701
I20260619 06:33:36.007578 29987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:36701 every 8 connection(s)
I20260619 06:33:36.008919 29860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:33:36.013695 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 29860
I20260619 06:33:36.013873 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:33:36.014458 29091 external_mini_cluster.cc:1428] Setting key 17490faada3fe50226f4109fb674e713
Jun 19 06:33:36 dist-test-slave-nf4n krb5kdc[29620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850815, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:36.030491 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.011575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:44525 (local address 127.28.104.254:35365)
0619 06:33:36.011760 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.011764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.012931 (+ 1167us) server_negotiation.cc:407] Connection header received
0619 06:33:36.015494 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.015500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.015603 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.015716 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:36.017623 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.018450 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.020483 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.020955 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.021142 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.024709 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:36.024714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:36.024717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:36.024748 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:36.027408 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.028162 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.028168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.028172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.028303 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.028723 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.028730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.028735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.029061 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:36.029226 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.029923 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.030153 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
I20260619 06:33:36.031577 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.012074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35365 (local address 127.28.104.194:44525)
0619 06:33:36.012766 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:36.012809 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:36.015227 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:36.015949 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:36.015958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:36.016583 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:36.017448 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.017473 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.018679 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.018684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.020318 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.020334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.021203 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.021211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.021453 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.022179 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:36.022201 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:36.024491 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:36.027561 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:36.027570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:36.027588 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:36.028020 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:36.028414 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:36.028420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:36.028423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:36.028616 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:36.029195 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:36.029203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:36.029555 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 06:33:36.029821 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 06:33:36.030378 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":256,"threads_started":1}
I20260619 06:33:36.033128 29988 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
I20260619 06:33:36.033483 29988 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:36.034425 29988 heartbeater.cc:511] Master 127.28.104.254:35365 requested a full tablet report, sending...
I20260619 06:33:36.036234 29665 ts_manager.cc:194] Registered new tserver with Master: 5c85e161085b497b9127b660f2433c83 (127.28.104.194:36701)
I20260619 06:33:36.037411 29665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:44525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:33:36.056167 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35365
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:36.386171 29993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:36.386456 29993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:36.386623 29993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:36.394160 29993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:36.394302 29993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:36.394490 29993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:33:36.404809 29993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:36.406201 29993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:36.407531 29993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:36.421392 29998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:36.421411 30001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:36.421458 29999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:36.422211 29993 server_base.cc:1034] running on GCE node
I20260619 06:33:36.422845 29993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:36.423869 29993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:36.425122 29993 hybrid_clock.cc:648] HybridClock initialized: now 1781850816425003 us; error 97 us; skew 500 ppm
I20260619 06:33:36.425504 29993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:36 dist-test-slave-nf4n krb5kdc[29620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850816, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:36.428676 29993 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:33:36.430215 29993 webserver.cc:460] Webserver started at http://127.28.104.195:43711/ using document root <none> and password file <none>
I20260619 06:33:36.430902 29993 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:36.431046 29993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:36.431366 29993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:36.434422 29993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "7ace1140ed2443baa3cea177fdb0623c"
format_stamp: "Formatted at 2026-06-19 06:33:36 on dist-test-slave-nf4n"
server_key: "e5ce9ba1a46c4cd0ac4f528263ae9e71"
server_key_iv: "b89f5ddfa02083e9e1181b962e389558"
server_key_version: "encryptionkey@0"
I20260619 06:33:36.435240 29993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "7ace1140ed2443baa3cea177fdb0623c"
format_stamp: "Formatted at 2026-06-19 06:33:36 on dist-test-slave-nf4n"
server_key: "e5ce9ba1a46c4cd0ac4f528263ae9e71"
server_key_iv: "b89f5ddfa02083e9e1181b962e389558"
server_key_version: "encryptionkey@0"
I20260619 06:33:36.440645 29993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 06:33:36.444409 30008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:36.445787 29993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:36.446030 29993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "7ace1140ed2443baa3cea177fdb0623c"
format_stamp: "Formatted at 2026-06-19 06:33:36 on dist-test-slave-nf4n"
server_key: "e5ce9ba1a46c4cd0ac4f528263ae9e71"
server_key_iv: "b89f5ddfa02083e9e1181b962e389558"
server_key_version: "encryptionkey@0"
I20260619 06:33:36.446260 29993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:36.462083 29993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:36.467070 29993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:36.467417 29993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:36.468477 29993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:36.470001 29993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:36.470114 29993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:36.470274 29993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:36.470358 29993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:36.503527 29993 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:38021
I20260619 06:33:36.503575 30120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:38021 every 8 connection(s)
I20260619 06:33:36.504944 29993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:33:36.513564 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 29993
I20260619 06:33:36.513743 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:33:36.514174 29091 external_mini_cluster.cc:1428] Setting key cfe4b18b8e4666fa866578a84984b45b
Jun 19 06:33:36 dist-test-slave-nf4n krb5kdc[29620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850816, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:36.525427 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.507770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:38777 (local address 127.28.104.254:35365)
0619 06:33:36.507984 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.507990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.509256 (+ 1266us) server_negotiation.cc:407] Connection header received
0619 06:33:36.511564 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.511568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.511640 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.511729 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:36.513365 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.514208 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.516100 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.516554 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.516654 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.520092 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:36.520097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:36.520100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:36.520131 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:36.522801 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.523421 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.523425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.523427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.523560 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.523912 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.523915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.523917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.524210 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:36.524320 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.524962 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.525184 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:33:36.526376 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.508251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35365 (local address 127.28.104.195:38777)
0619 06:33:36.509076 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:36.509124 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:36.511304 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:36.511930 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:36.511938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:36.512501 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:36.513192 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.513217 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.514428 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.514432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.515976 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.515988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.516776 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.516785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.517111 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.517756 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:36.517777 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:36.519916 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:36.522909 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:36.522915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:36.522928 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:36.523319 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:36.523651 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:36.523654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:36.523656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:36.523819 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:36.524314 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:36.524319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:36.524664 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 06:33:36.524876 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 06:33:36.525295 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":286,"threads_started":1}
I20260619 06:33:36.527945 30121 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35365
I20260619 06:33:36.528258 30121 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:36.529093 30121 heartbeater.cc:511] Master 127.28.104.254:35365 requested a full tablet report, sending...
I20260619 06:33:36.530616 29665 ts_manager.cc:194] Registered new tserver with Master: 7ace1140ed2443baa3cea177fdb0623c (127.28.104.195:38021)
I20260619 06:33:36.531462 29665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:38777
I20260619 06:33:36.539798 29854 heartbeater.cc:503] Master 127.28.104.254:35365 was elected leader, sending a full tablet report...
I20260619 06:33:36.542109 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:33:36.561863 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.548671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.28.104.254:35365)
0619 06:33:36.548843 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.548848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.549413 (+ 565us) server_negotiation.cc:407] Connection header received
0619 06:33:36.549643 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.549647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.549711 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.549815 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:36.551326 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.552104 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.553459 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.553883 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.554021 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.556420 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:36.556431 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:36.556434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:36.556467 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:36.559011 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.559657 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.559664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.559668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.559809 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.560309 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.560317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.560321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.560661 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:36.560911 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.561319 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.561533 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:33:36.568989 29664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:33:36.576538 29664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:33:36.616402 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.589523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:36701 (local address 127.0.0.1:41534)
0619 06:33:36.602696 (+ 13173us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:36.602768 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:36.602998 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:36.603953 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:36.603966 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:36.604007 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:36.604591 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.604619 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.607466 (+ 2847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.607476 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.610994 (+ 3518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.611017 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.613763 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.613773 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.614091 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.614173 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 06:33:36.614315 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 06:33:36.615787 (+ 1472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12897,"mutex_wait_us":96,"thread_start_us":12781,"threads_started":1}
I20260619 06:33:36.616513 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.589621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.28.104.193:38023)
0619 06:33:36.602477 (+ 12856us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.602492 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.603137 (+ 645us) server_negotiation.cc:407] Connection header received
0619 06:33:36.603395 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.603406 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.603676 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.603881 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:36.605257 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.607540 (+ 2283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.611517 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.613855 (+ 2338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.614144 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.614412 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.615676 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.615973 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12043,"thread_start_us":11823,"threads_started":1}
I20260619 06:33:36.617654 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.589615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:38023 (local address 127.0.0.1:43622)
0619 06:33:36.603004 (+ 13389us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:36.603033 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:36.603178 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:36.604263 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:36.604272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:36.604311 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:36.604958 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.604976 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.607797 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.607812 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.611228 (+ 3416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.611261 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.614611 (+ 3350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.614622 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.615322 (+ 700us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.615385 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 06:33:36.615495 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 06:33:36.616993 (+ 1498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13259,"mutex_wait_us":1243,"thread_start_us":13111,"threads_started":1}
I20260619 06:33:36.618495 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.589754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.28.104.194:36701)
0619 06:33:36.602269 (+ 12515us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.602281 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.603207 (+ 926us) server_negotiation.cc:407] Connection header received
0619 06:33:36.603320 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.603329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.603657 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.604857 (+ 1200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:36.605204 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.607273 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.611247 (+ 3974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.613441 (+ 2194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.617312 (+ 3871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.617564 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.617710 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.618044 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12312,"thread_start_us":12193,"threads_started":1}
I20260619 06:33:36.622605 29789 tablet_service.cc:1467] Processing CreateTablet for tablet 694ffe1f619b43399f4847b670feb17b (DEFAULT_TABLE table=test-table [id=28714b811bde445981623661f79b913d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:36.624219 29789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694ffe1f619b43399f4847b670feb17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:36.623623 29923 tablet_service.cc:1467] Processing CreateTablet for tablet 694ffe1f619b43399f4847b670feb17b (DEFAULT_TABLE table=test-table [id=28714b811bde445981623661f79b913d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:36.625240 29923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694ffe1f619b43399f4847b670feb17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:36.626343 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.603333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:38021 (local address 127.0.0.1:60380)
0619 06:33:36.616880 (+ 13547us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:36.616905 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:36.617064 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:36.619695 (+ 2631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:36.619700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:36.619729 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:36.619907 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.619919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.621943 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.621948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.623725 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.623743 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.625494 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.625501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.625707 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.625744 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:33:36.625849 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:33:36.625998 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13399,"thread_start_us":15700,"threads_started":1}
I20260619 06:33:36.626420 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.603706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.28.104.195:38021)
0619 06:33:36.619007 (+ 15301us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.619016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.619052 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:33:36.619164 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.619172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.619473 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.619668 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:36.620101 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.621781 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.623924 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.625351 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.625535 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.625713 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.625924 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.626071 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15097,"thread_start_us":15496,"threads_started":1}
I20260619 06:33:36.630458 30056 tablet_service.cc:1467] Processing CreateTablet for tablet 694ffe1f619b43399f4847b670feb17b (DEFAULT_TABLE table=test-table [id=28714b811bde445981623661f79b913d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:36.631915 30056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694ffe1f619b43399f4847b670feb17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:36.639513 30139 tablet_bootstrap.cc:492] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83: Bootstrap starting.
I20260619 06:33:36.641827 30140 tablet_bootstrap.cc:492] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809: Bootstrap starting.
I20260619 06:33:36.644187 30139 tablet_bootstrap.cc:654] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:36.645772 30140 tablet_bootstrap.cc:654] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:36.645774 30139 log.cc:824] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:36.647181 30140 log.cc:824] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:36.649248 30142 tablet_bootstrap.cc:492] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c: Bootstrap starting.
I20260619 06:33:36.649590 30139 tablet_bootstrap.cc:492] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83: No bootstrap required, opened a new log
I20260619 06:33:36.649881 30139 ts_tablet_manager.cc:1397] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260619 06:33:36.652532 30140 tablet_bootstrap.cc:492] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809: No bootstrap required, opened a new log
I20260619 06:33:36.652848 30140 ts_tablet_manager.cc:1397] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260619 06:33:36.653365 30142 tablet_bootstrap.cc:654] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:36.654333 30142 log.cc:824] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:36.657166 30142 tablet_bootstrap.cc:492] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c: No bootstrap required, opened a new log
I20260619 06:33:36.657441 30142 ts_tablet_manager.cc:1397] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 06:33:36.662091 30139 raft_consensus.cc:348] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b202f5428a124f1489ed3b723d663809" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38023 } } peers { permanent_uuid: "5c85e161085b497b9127b660f2433c83" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36701 } } peers { permanent_uuid: "7ace1140ed2443baa3cea177fdb0623c" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38021 } }
I20260619 06:33:36.662626 30139 raft_consensus.cc:374] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:36.662734 30139 raft_consensus.cc:729] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c85e161085b497b9127b660f2433c83, State: Initialized, Role: FOLLOWER
I20260619 06:33:36.663479 30139 consensus_queue.cc:260] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b202f5428a124f1489ed3b723d663809" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38023 } } peers { permanent_uuid: "5c85e161085b497b9127b660f2433c83" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36701 } } peers { permanent_uuid: "7ace1140ed2443baa3cea177fdb0623c" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38021 } }
I20260619 06:33:36.664705 29988 heartbeater.cc:503] Master 127.28.104.254:35365 was elected leader, sending a full tablet report...
I20260619 06:33:36.664654 30140 raft_consensus.cc:348] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b202f5428a124f1489ed3b723d663809" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38023 } } peers { permanent_uuid: "5c85e161085b497b9127b660f2433c83" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36701 } } peers { permanent_uuid: "7ace1140ed2443baa3cea177fdb0623c" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38021 } }
I20260619 06:33:36.665161 30140 raft_consensus.cc:374] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:36.665266 30140 raft_consensus.cc:729] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b202f5428a124f1489ed3b723d663809, State: Initialized, Role: FOLLOWER
I20260619 06:33:36.665553 30139 ts_tablet_manager.cc:1428] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260619 06:33:36.666035 30140 consensus_queue.cc:260] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b202f5428a124f1489ed3b723d663809" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38023 } } peers { permanent_uuid: "5c85e161085b497b9127b660f2433c83" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36701 } } peers { permanent_uuid: "7ace1140ed2443baa3cea177fdb0623c" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38021 } }
I20260619 06:33:36.666147 30142 raft_consensus.cc:348] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b202f5428a124f1489ed3b723d663809" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38023 } } peers { permanent_uuid: "5c85e161085b497b9127b660f2433c83" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36701 } } peers { permanent_uuid: "7ace1140ed2443baa3cea177fdb0623c" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38021 } }
I20260619 06:33:36.666571 30142 raft_consensus.cc:374] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:36.666697 30142 raft_consensus.cc:729] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ace1140ed2443baa3cea177fdb0623c, State: Initialized, Role: FOLLOWER
I20260619 06:33:36.667255 30142 consensus_queue.cc:260] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b202f5428a124f1489ed3b723d663809" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38023 } } peers { permanent_uuid: "5c85e161085b497b9127b660f2433c83" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36701 } } peers { permanent_uuid: "7ace1140ed2443baa3cea177fdb0623c" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38021 } }
I20260619 06:33:36.668285 30121 heartbeater.cc:503] Master 127.28.104.254:35365 was elected leader, sending a full tablet report...
I20260619 06:33:36.668962 30142 ts_tablet_manager.cc:1428] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 06:33:36.672770 30140 ts_tablet_manager.cc:1428] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260619 06:33:36.741686 30147 raft_consensus.cc:482] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:33:36.742074 30147 raft_consensus.cc:504] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b202f5428a124f1489ed3b723d663809" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38023 } } peers { permanent_uuid: "5c85e161085b497b9127b660f2433c83" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36701 } } peers { permanent_uuid: "7ace1140ed2443baa3cea177fdb0623c" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38021 } }
I20260619 06:33:36.744151 30147 leader_election.cc:288] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b202f5428a124f1489ed3b723d663809 (127.28.104.193:38023), 5c85e161085b497b9127b660f2433c83 (127.28.104.194:36701)
I20260619 06:33:36.749923 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.744431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:36701 (local address 127.28.104.195:39891)
0619 06:33:36.744868 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:36.744890 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:36.745097 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:36.745562 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:36.745567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:36.745607 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:36.745811 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.745820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.746840 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.746846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.748092 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.748113 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.749212 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.749218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.749400 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.749424 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:33:36.749529 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:33:36.749654 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260619 06:33:36.749955 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.744989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:39891 (local address 127.28.104.194:36701)
0619 06:33:36.745212 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.745217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.745235 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:33:36.745311 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.745318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.745407 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.745517 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:36.745962 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.746682 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.748270 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.749063 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.749227 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.749302 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.749554 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.749724 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 06:33:36.751071 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.744641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:59121 (local address 127.28.104.193:38023)
0619 06:33:36.744932 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.744941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.745876 (+ 935us) server_negotiation.cc:407] Connection header received
0619 06:33:36.746078 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.746082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.746143 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.746253 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:36.746754 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.747428 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.749062 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.750020 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.750185 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.750245 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.750666 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.750829 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 06:33:36.751091 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.744476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:38023 (local address 127.28.104.195:59121)
0619 06:33:36.745762 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:36.745779 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:36.745914 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:36.746266 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:36.746271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:36.746292 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:36.746600 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.746608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.747575 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.747581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.748880 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:36.748896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.750390 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.750394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:36.750504 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.750534 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:33:36.750640 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:33:36.750757 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"thread_start_us":927,"threads_started":1}
I20260619 06:33:36.751044 29943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694ffe1f619b43399f4847b670feb17b" candidate_uuid: "7ace1140ed2443baa3cea177fdb0623c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c85e161085b497b9127b660f2433c83" is_pre_election: true
I20260619 06:33:36.751636 29943 raft_consensus.cc:2454] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ace1140ed2443baa3cea177fdb0623c in term 0.
I20260619 06:33:36.752012 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694ffe1f619b43399f4847b670feb17b" candidate_uuid: "7ace1140ed2443baa3cea177fdb0623c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b202f5428a124f1489ed3b723d663809" is_pre_election: true
I20260619 06:33:36.752450 30011 leader_election.cc:302] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c85e161085b497b9127b660f2433c83, 7ace1140ed2443baa3cea177fdb0623c; no voters:
I20260619 06:33:36.752583 29809 raft_consensus.cc:2454] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ace1140ed2443baa3cea177fdb0623c in term 0.
I20260619 06:33:36.752935 30147 raft_consensus.cc:2781] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:33:36.753206 30147 raft_consensus.cc:482] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:33:36.753348 30147 raft_consensus.cc:3037] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:36.755560 30147 raft_consensus.cc:504] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b202f5428a124f1489ed3b723d663809" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38023 } } peers { permanent_uuid: "5c85e161085b497b9127b660f2433c83" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36701 } } peers { permanent_uuid: "7ace1140ed2443baa3cea177fdb0623c" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38021 } }
I20260619 06:33:36.756268 30147 leader_election.cc:288] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [CANDIDATE]: Term 1 election: Requested vote from peers b202f5428a124f1489ed3b723d663809 (127.28.104.193:38023), 5c85e161085b497b9127b660f2433c83 (127.28.104.194:36701)
I20260619 06:33:36.756641 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694ffe1f619b43399f4847b670feb17b" candidate_uuid: "7ace1140ed2443baa3cea177fdb0623c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b202f5428a124f1489ed3b723d663809"
W20260619 06:33:36.756798 30122 tablet.cc:2369] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:33:36.756745 29943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694ffe1f619b43399f4847b670feb17b" candidate_uuid: "7ace1140ed2443baa3cea177fdb0623c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c85e161085b497b9127b660f2433c83"
I20260619 06:33:36.756841 29809 raft_consensus.cc:3037] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:36.756971 29943 raft_consensus.cc:3037] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83 [term 0 FOLLOWER]: Advancing to term 1
W20260619 06:33:36.759131 29855 tablet.cc:2369] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:33:36.759202 29809 raft_consensus.cc:2454] T 694ffe1f619b43399f4847b670feb17b P b202f5428a124f1489ed3b723d663809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ace1140ed2443baa3cea177fdb0623c in term 1.
I20260619 06:33:36.759546 29943 raft_consensus.cc:2454] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ace1140ed2443baa3cea177fdb0623c in term 1.
I20260619 06:33:36.759645 30010 leader_election.cc:302] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ace1140ed2443baa3cea177fdb0623c, b202f5428a124f1489ed3b723d663809; no voters:
I20260619 06:33:36.759976 30147 raft_consensus.cc:2781] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:36.760418 30147 raft_consensus.cc:686] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [term 1 LEADER]: Becoming Leader. State: Replica: 7ace1140ed2443baa3cea177fdb0623c, State: Running, Role: LEADER
W20260619 06:33:36.760785 29989 tablet.cc:2369] T 694ffe1f619b43399f4847b670feb17b P 5c85e161085b497b9127b660f2433c83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:33:36.760912 30147 consensus_queue.cc:237] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b202f5428a124f1489ed3b723d663809" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38023 } } peers { permanent_uuid: "5c85e161085b497b9127b660f2433c83" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36701 } } peers { permanent_uuid: "7ace1140ed2443baa3cea177fdb0623c" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38021 } }
I20260619 06:33:36.765363 29664 catalog_manager.cc:5495] T 694ffe1f619b43399f4847b670feb17b P 7ace1140ed2443baa3cea177fdb0623c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ace1140ed2443baa3cea177fdb0623c (127.28.104.195). New cstate: current_term: 1 leader_uuid: "7ace1140ed2443baa3cea177fdb0623c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b202f5428a124f1489ed3b723d663809" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c85e161085b497b9127b660f2433c83" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ace1140ed2443baa3cea177fdb0623c" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38021 } health_report { overall_health: HEALTHY } } }
Jun 19 06:33:36 dist-test-slave-nf4n krb5kdc[29620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:33:36.796073 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.778320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.28.104.193:38023)
0619 06:33:36.778527 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.778536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.779482 (+ 946us) server_negotiation.cc:407] Connection header received
0619 06:33:36.779655 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.779662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.779752 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.779894 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:36.781824 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.783165 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.784962 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.785530 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.785643 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.789683 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:36.789699 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:36.789715 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:36.789752 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:36.793145 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.793814 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.793823 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.793825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.793963 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.794373 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.794377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.794380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.794814 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:36.795132 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.795426 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.795613 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 06:33:36 dist-test-slave-nf4n krb5kdc[29620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:33:36 dist-test-slave-nf4n krb5kdc[29620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:33:36.841491 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.827465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.28.104.193:38023)
0619 06:33:36.827765 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.827773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.828039 (+ 266us) server_negotiation.cc:407] Connection header received
0619 06:33:36.828284 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.828288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.828354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.828463 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:36.829863 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.830601 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.831885 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.832287 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.832426 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.836386 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:36.836391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:36.836395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:36.836429 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:36.838975 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.839632 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.839636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.839638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.839763 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.840182 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.840187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.840190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.840554 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:36.840769 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.841068 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.841246 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
Jun 19 06:33:36 dist-test-slave-nf4n krb5kdc[29620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:33:36 dist-test-slave-nf4n krb5kdc[29620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:33:36.887771 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.873438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.28.104.193:38023)
0619 06:33:36.873687 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.873694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.873872 (+ 178us) server_negotiation.cc:407] Connection header received
0619 06:33:36.874093 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.874097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.874159 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.874268 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:36.875624 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.876299 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.877621 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.878043 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.878183 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.882002 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:36.882009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:36.882015 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:36.882056 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:36.885329 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.885947 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.885952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.885956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.886100 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.886414 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.886418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.886422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.886773 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:36.887017 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.887205 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.887426 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 06:33:36.889732 29769 tablet_service.cc:2908] Rejecting scan request for tablet 694ffe1f619b43399f4847b670feb17b: Uninitialized: clean time has not yet been initialized
I20260619 06:33:36.893098 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 29726
I20260619 06:33:36.905642 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:36.894385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.28.104.193:38023)
0619 06:33:36.894525 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:36.894529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:36.894954 (+ 425us) server_negotiation.cc:407] Connection header received
0619 06:33:36.895073 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:36.895076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:36.895176 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:36.895255 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:36.896307 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.896952 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.898127 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:36.898504 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:36.898618 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:36.900719 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:36.900723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:36.900727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:36.900752 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:36.903219 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.903836 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.903839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.903842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.903956 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:36.904254 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:36.904257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:36.904259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:36.904557 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:36.904756 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:36.905139 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 06:33:36.905300 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:33:36.906266 29849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:36.996318 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 29726
I20260619 06:33:37.013795 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 29860
Jun 19 06:33:37 dist-test-slave-nf4n krb5kdc[29620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:33:37.028769 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:37.014600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.28.104.194:36701)
0619 06:33:37.014793 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:37.014799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:37.014823 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:33:37.014991 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:37.014997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:37.015067 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:37.015183 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:37.016486 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:37.017368 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:37.018712 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:37.019107 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:37.019241 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:37.023324 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:37.023334 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:37.023348 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:37.023379 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:37.026211 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:37.026881 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:37.026888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:37.026890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:37.027008 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:37.027369 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:37.027372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:37.027374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:37.027719 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:37.027931 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:37.028252 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 06:33:37.028411 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:33:37.029593 29983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:37.119278 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 29860
I20260619 06:33:37.137306 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 29993
Jun 19 06:33:37 dist-test-slave-nf4n krb5kdc[29620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:33:37.152388 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:37.138100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.28.104.195:38021)
0619 06:33:37.138567 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:37.138571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:37.138590 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:33:37.138650 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:37.138655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:37.138719 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:37.138825 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:37.140006 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:37.140801 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:37.142086 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:37.142505 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:37.142610 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:37.146663 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:37.146678 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:37.146694 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:37.146731 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:37.149849 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:37.150534 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:37.150541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:37.150543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:37.150665 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:37.151045 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:37.151048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:37.151050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:37.151389 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:37.151611 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:37.151799 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 06:33:37.151977 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":245,"threads_started":1}
I20260619 06:33:37.153149 30116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:37.243448 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 29993
I20260619 06:33:37.260236 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 29633
I20260619 06:33:37.261466 29695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:37.368021 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 29633
2026-06-19T06:33:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7756 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:33:37 dist-test-slave-nf4n krb5kdc[30169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:33:37 dist-test-slave-nf4n krb5kdc[30169](info): set up 2 sockets
Jun 19 06:33:37 dist-test-slave-nf4n krb5kdc[30169](info): commencing operation
krb5kdc: starting...
W20260619 06:33:38.488853 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:33:38 dist-test-slave-nf4n krb5kdc[30169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:33:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:33:38.670732 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:35503
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:35503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:38.992287 30182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:38.992573 30182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:38.992765 30182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:39.000200 30182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:39.000308 30182 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:33:39.000386 30182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:39.000497 30182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:39.000579 30182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:39.000667 30182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:39.010699 30182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:35503
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:35503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:39.012058 30182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:39.013449 30182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:39.025157 30187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:39.025130 30188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:39.025184 30190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:39.026387 30182 server_base.cc:1034] running on GCE node
I20260619 06:33:39.027180 30182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:39.028620 30182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:39.029855 30182 hybrid_clock.cc:648] HybridClock initialized: now 1781850819029785 us; error 53 us; skew 500 ppm
I20260619 06:33:39.030213 30182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:39 dist-test-slave-nf4n krb5kdc[30169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850819, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:39.033396 30182 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:33:39.034861 30182 webserver.cc:460] Webserver started at http://127.28.104.254:44089/ using document root <none> and password file <none>
I20260619 06:33:39.035538 30182 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:39.035622 30182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:39.035876 30182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:39.038851 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b"
format_stamp: "Formatted at 2026-06-19 06:33:39 on dist-test-slave-nf4n"
server_key: "e35c2e6c5e52a35f8ce8af70f0094716"
server_key_iv: "9717ddf1105b9e95a3ee236fb64866c3"
server_key_version: "encryptionkey@0"
I20260619 06:33:39.039564 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b"
format_stamp: "Formatted at 2026-06-19 06:33:39 on dist-test-slave-nf4n"
server_key: "e35c2e6c5e52a35f8ce8af70f0094716"
server_key_iv: "9717ddf1105b9e95a3ee236fb64866c3"
server_key_version: "encryptionkey@0"
I20260619 06:33:39.044739 30182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:33:39.048661 30197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:39.050088 30182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:33:39.050273 30182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b"
format_stamp: "Formatted at 2026-06-19 06:33:39 on dist-test-slave-nf4n"
server_key: "e35c2e6c5e52a35f8ce8af70f0094716"
server_key_iv: "9717ddf1105b9e95a3ee236fb64866c3"
server_key_version: "encryptionkey@0"
I20260619 06:33:39.050493 30182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:39.063724 30182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:39.067948 30182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:39.068229 30182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:39.090435 30182 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:35503
I20260619 06:33:39.090446 30258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:35503 every 8 connection(s)
I20260619 06:33:39.091840 30182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:33:39.095678 30259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:39.098019 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 30182
I20260619 06:33:39.098227 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:33:39.099063 29091 external_mini_cluster.cc:1428] Setting key c976044674788975a6c2855ada236d3c
I20260619 06:33:39.108311 30259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b: Bootstrap starting.
I20260619 06:33:39.112280 30259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:39.113569 30259 log.cc:824] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:33:39 dist-test-slave-nf4n krb5kdc[30169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:39.116511 30259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b: No bootstrap required, opened a new log
I20260619 06:33:39.122220 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:39.101158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.28.104.254:35503)
0619 06:33:39.101961 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:39.101973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:39.102015 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:33:39.103679 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:39.103702 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:39.104067 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:39.104402 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:39.106142 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:39.107411 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:39.109184 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:39.109836 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:39.109973 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:39.114976 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:39.114991 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:39.115012 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:39.115060 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:39.118129 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:39.118827 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:39.118834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:39.118839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:39.119014 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:39.119418 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:39.119422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:39.119424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:39.119946 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:39.120262 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:39.120720 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 06:33:39.121195 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":383,"threads_started":1}
I20260619 06:33:39.124269 30259 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35503 } }
I20260619 06:33:39.124641 30259 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:39.124771 30259 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0dc9a3c15c9473ca697c802b4ea0d6b, State: Initialized, Role: FOLLOWER
I20260619 06:33:39.125439 30259 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35503 } }
I20260619 06:33:39.125656 30259 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:33:39.125757 30259 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:33:39.125874 30259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:39.128700 30259 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35503 } }
I20260619 06:33:39.129405 30259 leader_election.cc:302] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0dc9a3c15c9473ca697c802b4ea0d6b; no voters:
I20260619 06:33:39.130128 30259 leader_election.cc:288] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:33:39.130260 30264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:39.131182 30264 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [term 1 LEADER]: Becoming Leader. State: Replica: a0dc9a3c15c9473ca697c802b4ea0d6b, State: Running, Role: LEADER
I20260619 06:33:39.131862 30264 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35503 } }
I20260619 06:33:39.132380 30259 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:39.134526 30265 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35503 } } }
I20260619 06:33:39.134896 30265 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:39.135329 30266 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0dc9a3c15c9473ca697c802b4ea0d6b. Latest consensus state: current_term: 1 leader_uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dc9a3c15c9473ca697c802b4ea0d6b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35503 } } }
I20260619 06:33:39.135604 30266 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:39.137681 30271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:33:39.143281 30271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:33:39.149924 30281 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:33:39.154664 30271 catalog_manager.cc:1347] Generated new cluster ID: 1379f03ea4ac487784d186058f8cefe3
I20260619 06:33:39.154783 30271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:33:39.161620 30271 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:33:39.163517 30271 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:33:39.176438 30271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b: Generated new TSK 0
I20260619 06:33:39.177484 30271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:33:39.224776 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35503
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:39.548828 30285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:39.549186 30285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:39.549358 30285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:39.557139 30285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:39.557257 30285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:39.557431 30285 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:33:39.557549 30285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:33:39.567723 30285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.104.254:35503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:39.569159 30285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:39.570592 30285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:39.583935 30290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:39.583969 30291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:39.584010 30293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:39.584726 30285 server_base.cc:1034] running on GCE node
I20260619 06:33:39.585366 30285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:39.586236 30285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:39.587440 30285 hybrid_clock.cc:648] HybridClock initialized: now 1781850819587360 us; error 57 us; skew 500 ppm
I20260619 06:33:39.587843 30285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:39 dist-test-slave-nf4n krb5kdc[30169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850819, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:39.591168 30285 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:33:39.592566 30285 webserver.cc:460] Webserver started at http://127.28.104.193:34395/ using document root <none> and password file <none>
I20260619 06:33:39.593346 30285 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:39.593437 30285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:39.593715 30285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:39.596503 30285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "20405f9eb2a14ef79a970e920c7b6f98"
format_stamp: "Formatted at 2026-06-19 06:33:39 on dist-test-slave-nf4n"
server_key: "4827e778eaaaaae3d719dde2f090824c"
server_key_iv: "0c8eccd5b177d4e1d0f6c4b728ef374d"
server_key_version: "encryptionkey@0"
I20260619 06:33:39.597321 30285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "20405f9eb2a14ef79a970e920c7b6f98"
format_stamp: "Formatted at 2026-06-19 06:33:39 on dist-test-slave-nf4n"
server_key: "4827e778eaaaaae3d719dde2f090824c"
server_key_iv: "0c8eccd5b177d4e1d0f6c4b728ef374d"
server_key_version: "encryptionkey@0"
I20260619 06:33:39.602475 30285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 06:33:39.605973 30300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:39.607299 30285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:33:39.607460 30285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "20405f9eb2a14ef79a970e920c7b6f98"
format_stamp: "Formatted at 2026-06-19 06:33:39 on dist-test-slave-nf4n"
server_key: "4827e778eaaaaae3d719dde2f090824c"
server_key_iv: "0c8eccd5b177d4e1d0f6c4b728ef374d"
server_key_version: "encryptionkey@0"
I20260619 06:33:39.607659 30285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:39.622603 30285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:39.627115 30285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:39.627398 30285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:39.630149 30285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:39.630264 30285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:39.630410 30285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:39.630507 30285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:33:39 dist-test-slave-nf4n krb5kdc[30169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850819, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:39.658212 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:39.631090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:36995 (local address 127.28.104.254:35503)
0619 06:33:39.631772 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:39.631780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:39.633357 (+ 1577us) server_negotiation.cc:407] Connection header received
0619 06:33:39.636630 (+ 3273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:39.636637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:39.636744 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:39.636896 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:39.639573 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:39.640761 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:39.643933 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:39.644744 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:39.644919 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:39.650759 (+ 5840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:39.650767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:39.650772 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:39.650831 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:39.654176 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:39.655322 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:39.655329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:39.655333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:39.655515 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:39.656095 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:39.656101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:39.656105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:39.656532 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:39.656677 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:39.657615 (+ 938us) server_negotiation.cc:299] Negotiation successful
0619 06:33:39.657838 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":379,"threads_started":1}
I20260619 06:33:39.659302 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:39.631906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35503 (local address 127.28.104.193:36995)
0619 06:33:39.633160 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:39.633225 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:39.636230 (+ 3005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:39.637214 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:39.637224 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:39.638177 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:39.639336 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:39.639372 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:39.641071 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:39.641082 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:39.643631 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:39.643655 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:39.645082 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:39.645097 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:39.645570 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:39.646647 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:39.646675 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:39.650424 (+ 3749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:39.654350 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:39.654362 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:39.654387 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:39.655131 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:39.655700 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:39.655707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:39.655712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:39.655963 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:39.656660 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:39.656668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:39.657224 (+ 556us) client_negotiation.cc:769] Sending connection context
0619 06:33:39.657536 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 06:33:39.658088 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":624,"threads_started":1}
I20260619 06:33:39.681609 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:39.666416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:40697 (local address 127.28.104.254:35503)
0619 06:33:39.666581 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:39.666585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:39.667210 (+ 625us) server_negotiation.cc:407] Connection header received
0619 06:33:39.667412 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:39.667416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:39.667476 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:39.667597 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:39.669076 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:39.670002 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:39.672082 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:39.672721 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:39.672869 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:39.675103 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:39.675109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:39.675114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:39.675151 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:39.678699 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:39.679434 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:39.679441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:39.679445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:39.679638 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:39.680157 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:39.680160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:39.680162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:39.680421 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:39.680532 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:39.681205 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 06:33:39.681368 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:33:39.682345 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:39.666492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35503 (local address 127.28.104.193:40697)
0619 06:33:39.667103 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:39.667118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:39.667273 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:39.667788 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:39.667794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:39.668441 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:39.668835 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:39.668844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:39.670189 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:39.670197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:39.671906 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:39.671926 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:39.672928 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:39.672935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:39.673156 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:39.673870 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:39.673887 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:39.674922 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:39.678846 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:39.678851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:39.678856 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:39.679295 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:39.679788 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:39.679802 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:39.679809 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:39.680025 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:39.680545 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:39.680554 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:39.680848 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 06:33:39.681131 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 06:33:39.681621 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":346,"threads_started":1}
I20260619 06:33:39.692574 30285 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:46647
I20260619 06:33:39.692592 30420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:46647 every 8 connection(s)
I20260619 06:33:39.693998 30285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:33:39.696291 30421 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35503
I20260619 06:33:39.696532 30421 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:39.697290 30421 heartbeater.cc:511] Master 127.28.104.254:35503 requested a full tablet report, sending...
I20260619 06:33:39.699414 30214 ts_manager.cc:194] Registered new tserver with Master: 20405f9eb2a14ef79a970e920c7b6f98 (127.28.104.193:46647)
I20260619 06:33:39.700976 30214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:36995
I20260619 06:33:39.703940 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 30285
I20260619 06:33:39.704152 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:33:39.704782 29091 external_mini_cluster.cc:1428] Setting key 620dcd52c08080c9fd33f7c8dabaa866
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:33:39.744097 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35503
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:40.064420 30425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:40.064752 30425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:40.064919 30425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:40.072592 30425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:40.072767 30425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:40.072925 30425 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:33:40.073092 30425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:33:40.083001 30425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.104.254:35503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:40.084321 30425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:40.085719 30425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:40.099619 30431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:40.099920 30430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:40.100225 30433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:40.100246 30425 server_base.cc:1034] running on GCE node
I20260619 06:33:40.100991 30425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:40.102109 30425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:40.103368 30425 hybrid_clock.cc:648] HybridClock initialized: now 1781850820103273 us; error 58 us; skew 500 ppm
I20260619 06:33:40.103714 30425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:40 dist-test-slave-nf4n krb5kdc[30169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850820, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:40.107395 30425 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:33:40.108881 30425 webserver.cc:460] Webserver started at http://127.28.104.194:36239/ using document root <none> and password file <none>
I20260619 06:33:40.109642 30425 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:40.109730 30425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:40.110059 30425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:40.112982 30425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "f61e6ef921274d3aacf8e7d82eb43116"
format_stamp: "Formatted at 2026-06-19 06:33:40 on dist-test-slave-nf4n"
server_key: "daa662299ca3596e0176240ef435c3cc"
server_key_iv: "6f95b5bbc016a7d88458142fc2120d91"
server_key_version: "encryptionkey@0"
I20260619 06:33:40.113855 30425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "f61e6ef921274d3aacf8e7d82eb43116"
format_stamp: "Formatted at 2026-06-19 06:33:40 on dist-test-slave-nf4n"
server_key: "daa662299ca3596e0176240ef435c3cc"
server_key_iv: "6f95b5bbc016a7d88458142fc2120d91"
server_key_version: "encryptionkey@0"
I20260619 06:33:40.119231 30425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:33:40.122772 30440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:40.124023 30425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:33:40.124209 30425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "f61e6ef921274d3aacf8e7d82eb43116"
format_stamp: "Formatted at 2026-06-19 06:33:40 on dist-test-slave-nf4n"
server_key: "daa662299ca3596e0176240ef435c3cc"
server_key_iv: "6f95b5bbc016a7d88458142fc2120d91"
server_key_version: "encryptionkey@0"
I20260619 06:33:40.124413 30425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:40.159864 30425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:40.164374 30425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:40.164654 30425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:40.167512 30425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:40.167629 30425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:40.167783 30425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:40.167893 30425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:33:40 dist-test-slave-nf4n krb5kdc[30169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850820, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:40.196985 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.168687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:48413 (local address 127.28.104.254:35503)
0619 06:33:40.168923 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:40.168932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:40.170830 (+ 1898us) server_negotiation.cc:407] Connection header received
0619 06:33:40.174749 (+ 3919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:40.174758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:40.174878 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:40.175024 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:40.178061 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.179166 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.181148 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.181779 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.181886 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.188280 (+ 6394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:40.188292 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:40.188301 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:40.188364 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:40.192322 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:40.193745 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:40.193754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:40.193761 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:40.193952 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:40.194662 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:40.194669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:40.194673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:40.195200 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:40.195384 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:40.196246 (+ 862us) server_negotiation.cc:299] Negotiation successful
0619 06:33:40.196527 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:33:40.198927 30449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.169492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35503 (local address 127.28.104.194:48413)
0619 06:33:40.170602 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:40.170664 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:40.174193 (+ 3529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:40.175387 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:40.175396 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:40.176633 (+ 1237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:40.177730 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.177770 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.179376 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.179392 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.180894 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.180908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.182418 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.182436 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.182843 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.183941 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:40.183969 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:40.187971 (+ 4002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:40.192577 (+ 4606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:40.192595 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:40.192615 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:40.193481 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:40.194134 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:40.194140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:40.194145 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:40.194525 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:40.195372 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:40.195381 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:40.195801 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 06:33:40.196116 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 06:33:40.196735 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":525,"threads_started":1}
I20260619 06:33:40.223053 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.207340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:44247 (local address 127.28.104.254:35503)
0619 06:33:40.207562 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:40.207567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:40.208213 (+ 646us) server_negotiation.cc:407] Connection header received
0619 06:33:40.208454 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:40.208461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:40.208549 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:40.208676 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:40.210285 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.211600 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.213946 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.214437 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.214594 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.216734 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:40.216740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:40.216745 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:40.216785 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:40.219739 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:40.220698 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:40.220704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:40.220708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:40.220882 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:40.221429 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:40.221436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:40.221442 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:40.221886 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:40.222051 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:40.222456 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 06:33:40.222689 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 06:33:40.223214 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.207431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35503 (local address 127.28.104.194:44247)
0619 06:33:40.208064 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:40.208089 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:40.208291 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:40.208701 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:40.208707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:40.209342 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:40.209999 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.210014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.211902 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.211907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.213727 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.213751 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.214672 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.214680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.214893 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.215798 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:40.215815 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:40.216540 (+ 725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:40.219871 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:40.219879 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:40.219885 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:40.220508 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:40.221075 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:40.221082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:40.221087 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:40.221286 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:40.222021 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:40.222025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:40.222212 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 06:33:40.222354 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 06:33:40.222687 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":338,"threads_started":1}
I20260619 06:33:40.231276 30425 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:40495
I20260619 06:33:40.231298 30559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:40495 every 8 connection(s)
I20260619 06:33:40.232525 30425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:33:40.233678 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 30425
I20260619 06:33:40.233853 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:33:40.234457 29091 external_mini_cluster.cc:1428] Setting key f08c4803b68973442b5c0e24de1fe9e6
I20260619 06:33:40.234833 30560 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35503
I20260619 06:33:40.235677 30560 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:40.236429 30560 heartbeater.cc:511] Master 127.28.104.254:35503 requested a full tablet report, sending...
I20260619 06:33:40.237852 30214 ts_manager.cc:194] Registered new tserver with Master: f61e6ef921274d3aacf8e7d82eb43116 (127.28.104.194:40495)
I20260619 06:33:40.239137 30214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:48413
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:33:40.276404 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35503
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:40.595155 30564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:40.595486 30564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:40.595655 30564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:40.603117 30564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:40.603271 30564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:40.603425 30564 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:33:40.603560 30564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:33:40.613905 30564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.104.254:35503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:40.615257 30564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:40.616600 30564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:40.630188 30569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:40.630100 30572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:40.630064 30570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:40.631160 30564 server_base.cc:1034] running on GCE node
I20260619 06:33:40.631757 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:40.632702 30564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:40.634011 30564 hybrid_clock.cc:648] HybridClock initialized: now 1781850820633926 us; error 63 us; skew 500 ppm
I20260619 06:33:40.634317 30564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:40 dist-test-slave-nf4n krb5kdc[30169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850820, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:40.637439 30564 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:33:40.639062 30564 webserver.cc:460] Webserver started at http://127.28.104.195:35993/ using document root <none> and password file <none>
I20260619 06:33:40.639780 30564 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:40.639910 30564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:40.640240 30564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:40.643187 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "bc580958ba31435ea2cfd72c89c5a083"
format_stamp: "Formatted at 2026-06-19 06:33:40 on dist-test-slave-nf4n"
server_key: "fd3527c1bb6e302ae7111dac48916b77"
server_key_iv: "b4a0f18f2919dea26932f31bd3686559"
server_key_version: "encryptionkey@0"
I20260619 06:33:40.643991 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "bc580958ba31435ea2cfd72c89c5a083"
format_stamp: "Formatted at 2026-06-19 06:33:40 on dist-test-slave-nf4n"
server_key: "fd3527c1bb6e302ae7111dac48916b77"
server_key_iv: "b4a0f18f2919dea26932f31bd3686559"
server_key_version: "encryptionkey@0"
I20260619 06:33:40.649420 30564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260619 06:33:40.653357 30579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:40.654958 30564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:33:40.655128 30564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "bc580958ba31435ea2cfd72c89c5a083"
format_stamp: "Formatted at 2026-06-19 06:33:40 on dist-test-slave-nf4n"
server_key: "fd3527c1bb6e302ae7111dac48916b77"
server_key_iv: "b4a0f18f2919dea26932f31bd3686559"
server_key_version: "encryptionkey@0"
I20260619 06:33:40.655362 30564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:40.684165 30564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:40.688439 30564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:40.688750 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:40.691552 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:40.691689 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:40.691849 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:40.692075 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 06:33:40.703805 30421 heartbeater.cc:503] Master 127.28.104.254:35503 was elected leader, sending a full tablet report...
Jun 19 06:33:40 dist-test-slave-nf4n krb5kdc[30169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850820, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:40.717501 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.692108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:37803 (local address 127.28.104.254:35503)
0619 06:33:40.692299 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:40.692307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:40.693926 (+ 1619us) server_negotiation.cc:407] Connection header received
0619 06:33:40.696929 (+ 3003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:40.696935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:40.697046 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:40.697132 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:40.700274 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.701324 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.703438 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.704172 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.704320 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.709252 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:40.709261 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:40.709266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:40.709309 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:40.713000 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:40.714312 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:40.714319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:40.714324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:40.714481 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:40.715132 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:40.715135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:40.715137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:40.715415 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:40.715524 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:40.716751 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0619 06:33:40.716959 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:33:40.719362 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.692764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35503 (local address 127.28.104.195:37803)
0619 06:33:40.693749 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:40.693796 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:40.696550 (+ 2754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:40.697796 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:40.697809 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:40.698889 (+ 1080us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:40.700011 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.700044 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.701489 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.701496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.703198 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.703212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.704515 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.704527 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.704852 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.705691 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:40.705729 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:40.708931 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:40.713206 (+ 4275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:40.713218 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:40.713238 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:40.714142 (+ 904us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:40.714620 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:40.714629 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:40.714645 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:40.714998 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:40.715570 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:40.715582 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:40.716210 (+ 628us) client_negotiation.cc:769] Sending connection context
0619 06:33:40.716607 (+ 397us) client_negotiation.cc:240] Negotiation successful
0619 06:33:40.717375 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":402,"threads_started":1}
I20260619 06:33:40.741909 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.725962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:38161 (local address 127.28.104.254:35503)
0619 06:33:40.726189 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:40.726197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:40.726773 (+ 576us) server_negotiation.cc:407] Connection header received
0619 06:33:40.726936 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:40.726944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:40.727061 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:40.727196 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:40.728434 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.729738 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.732576 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.733316 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.733469 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.735761 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:40.735771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:40.735777 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:40.735827 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:40.739226 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:40.739890 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:40.739897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:40.739901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:40.740037 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:40.740403 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:40.740410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:40.740413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:40.740689 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:40.740851 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:40.741324 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 06:33:40.741541 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:33:40.742065 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.725922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35503 (local address 127.28.104.195:38161)
0619 06:33:40.726667 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:40.726691 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:40.726812 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:40.727227 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:40.727230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:40.727952 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:40.728285 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.728295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.729935 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.729944 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.732387 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.732408 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.733563 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.733572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.733842 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.734743 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:40.734761 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:40.735554 (+ 793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:40.739353 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:40.739359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:40.739363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:40.739754 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:40.740155 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:40.740160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:40.740164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:40.740296 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:40.740814 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:40.740818 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:40.741003 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 06:33:40.741219 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 06:33:40.741582 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":414,"threads_started":1}
I20260619 06:33:40.751926 30564 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:38509
I20260619 06:33:40.751950 30698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:38509 every 8 connection(s)
I20260619 06:33:40.753523 30564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:33:40.755527 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 30564
I20260619 06:33:40.755730 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:33:40.755796 30699 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35503
I20260619 06:33:40.755981 30699 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:40.756179 29091 external_mini_cluster.cc:1428] Setting key d71f0deb91441a00cd3b378662bb415d
I20260619 06:33:40.756572 30699 heartbeater.cc:511] Master 127.28.104.254:35503 requested a full tablet report, sending...
I20260619 06:33:40.758206 30214 ts_manager.cc:194] Registered new tserver with Master: bc580958ba31435ea2cfd72c89c5a083 (127.28.104.195:38509)
I20260619 06:33:40.759429 30214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:37803
I20260619 06:33:40.759701 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:33:40 dist-test-slave-nf4n krb5kdc[30169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:33:40 dist-test-slave-nf4n krb5kdc[30169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:40.808516 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.795199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.28.104.254:35503)
0619 06:33:40.795420 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:40.795427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:40.795621 (+ 194us) server_negotiation.cc:407] Connection header received
0619 06:33:40.795778 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:40.795782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:40.795847 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:40.795933 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:40.797428 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.798175 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.799426 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.799848 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.799939 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.803836 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:40.803841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:40.803844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:40.803877 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:40.806307 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:40.806882 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:40.806886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:40.806889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:40.806998 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:40.807317 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:40.807320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:40.807322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:40.807582 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:40.807763 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:40.807982 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 06:33:40.808142 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 06:33:40.813235 30214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:33:40.820781 30214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:33:40.856818 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.834676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.28.104.195:38509)
0619 06:33:40.849301 (+ 14625us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:40.849308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:40.849334 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:33:40.849408 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:40.849412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:40.849589 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:40.849802 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:40.850750 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.852248 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.854406 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.855555 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.855776 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.855975 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:40.856296 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 06:33:40.856471 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14465,"thread_start_us":14240,"threads_started":1}
I20260619 06:33:40.857786 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.834334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:38509 (local address 127.0.0.1:55938)
0619 06:33:40.835143 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:40.835198 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:40.835385 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:40.849811 (+ 14426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:40.849820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:40.849913 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:40.850376 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.850385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.852377 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.852385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.854254 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.854267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.855776 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.855782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.856010 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.856071 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 06:33:40.856226 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 06:33:40.856430 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":2,"spinlock_wait_cycles":85120,"thread_start_us":953,"threads_started":1}
I20260619 06:33:40.858798 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.835835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.28.104.194:40495)
0619 06:33:40.847669 (+ 11834us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:40.847680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:40.848611 (+ 931us) server_negotiation.cc:407] Connection header received
0619 06:33:40.848975 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:40.848985 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:40.849322 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:40.849524 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:40.851386 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.852842 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.856056 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.857453 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.857619 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.857808 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:40.858370 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 06:33:40.858537 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11634,"thread_start_us":11486,"threads_started":1}
I20260619 06:33:40.858819 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.835191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40495 (local address 127.0.0.1:46214)
0619 06:33:40.848462 (+ 13271us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:40.848486 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:40.848736 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:40.849810 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:40.849818 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:40.849887 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:40.851189 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.851201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.853053 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.853057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.855861 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.855871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.858015 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.858020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.858196 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.858219 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:33:40.858295 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:33:40.858452 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13093,"mutex_wait_us":238,"spinlock_wait_cycles":33024,"thread_start_us":13291,"threads_started":1}
I20260619 06:33:40.860131 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.836388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.28.104.193:46647)
0619 06:33:40.847157 (+ 10769us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:40.847171 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:40.848870 (+ 1699us) server_negotiation.cc:407] Connection header received
0619 06:33:40.849176 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:40.849183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:40.849370 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:40.849523 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:40.851218 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.852825 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.856087 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.857576 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.857782 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.857967 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:40.859642 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0619 06:33:40.859832 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10498,"thread_start_us":11305,"threads_started":1}
I20260619 06:33:40.860945 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:40.836058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:46647 (local address 127.0.0.1:34126)
0619 06:33:40.848756 (+ 12698us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:40.848778 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:40.848962 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:40.849810 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:40.849818 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:40.849902 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:40.851011 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.851025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.853050 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.853055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.855857 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:40.855871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:40.859214 (+ 3343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:40.859220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:40.859404 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:40.859435 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:33:40.860420 (+ 985us) client_negotiation.cc:240] Negotiation successful
0619 06:33:40.860562 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12586,"mutex_wait_us":194,"spinlock_wait_cycles":14848,"thread_start_us":12090,"threads_started":1}
I20260619 06:33:40.862286 30490 tablet_service.cc:1467] Processing CreateTablet for tablet 385644e8ad3a48f28818a8cf468d2906 (DEFAULT_TABLE table=test-table [id=92b271012fb7490e8e2570648964482e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:40.862305 30629 tablet_service.cc:1467] Processing CreateTablet for tablet 385644e8ad3a48f28818a8cf468d2906 (DEFAULT_TABLE table=test-table [id=92b271012fb7490e8e2570648964482e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:40.863448 30490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385644e8ad3a48f28818a8cf468d2906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:40.863689 30629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385644e8ad3a48f28818a8cf468d2906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:40.864440 30351 tablet_service.cc:1467] Processing CreateTablet for tablet 385644e8ad3a48f28818a8cf468d2906 (DEFAULT_TABLE table=test-table [id=92b271012fb7490e8e2570648964482e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:40.865806 30351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385644e8ad3a48f28818a8cf468d2906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:40.875437 30718 tablet_bootstrap.cc:492] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: Bootstrap starting.
I20260619 06:33:40.878319 30719 tablet_bootstrap.cc:492] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: Bootstrap starting.
I20260619 06:33:40.879153 30718 tablet_bootstrap.cc:654] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:40.879829 30720 tablet_bootstrap.cc:492] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: Bootstrap starting.
I20260619 06:33:40.880664 30718 log.cc:824] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:40.881861 30719 tablet_bootstrap.cc:654] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:40.882992 30720 tablet_bootstrap.cc:654] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:40.883157 30719 log.cc:824] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:40.884085 30718 tablet_bootstrap.cc:492] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: No bootstrap required, opened a new log
I20260619 06:33:40.884174 30720 log.cc:824] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:40.884420 30718 ts_tablet_manager.cc:1397] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 06:33:40.891887 30719 tablet_bootstrap.cc:492] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: No bootstrap required, opened a new log
I20260619 06:33:40.891914 30720 tablet_bootstrap.cc:492] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: No bootstrap required, opened a new log
I20260619 06:33:40.892187 30720 ts_tablet_manager.cc:1397] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260619 06:33:40.892228 30719 ts_tablet_manager.cc:1397] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260619 06:33:40.894935 30718 raft_consensus.cc:348] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } }
I20260619 06:33:40.895339 30718 raft_consensus.cc:374] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:40.895439 30718 raft_consensus.cc:729] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61e6ef921274d3aacf8e7d82eb43116, State: Initialized, Role: FOLLOWER
I20260619 06:33:40.896085 30718 consensus_queue.cc:260] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } }
I20260619 06:33:40.897223 30560 heartbeater.cc:503] Master 127.28.104.254:35503 was elected leader, sending a full tablet report...
I20260619 06:33:40.897908 30718 ts_tablet_manager.cc:1428] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: Time spent starting tablet: real 0.013s user 0.009s sys 0.006s
I20260619 06:33:40.903010 30719 raft_consensus.cc:348] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } }
I20260619 06:33:40.903061 30720 raft_consensus.cc:348] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } }
I20260619 06:33:40.903491 30720 raft_consensus.cc:374] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:40.903494 30719 raft_consensus.cc:374] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:40.903596 30719 raft_consensus.cc:729] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc580958ba31435ea2cfd72c89c5a083, State: Initialized, Role: FOLLOWER
I20260619 06:33:40.903606 30720 raft_consensus.cc:729] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20405f9eb2a14ef79a970e920c7b6f98, State: Initialized, Role: FOLLOWER
I20260619 06:33:40.904232 30719 consensus_queue.cc:260] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } }
I20260619 06:33:40.904232 30720 consensus_queue.cc:260] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } }
I20260619 06:33:40.905342 30699 heartbeater.cc:503] Master 127.28.104.254:35503 was elected leader, sending a full tablet report...
I20260619 06:33:40.906136 30719 ts_tablet_manager.cc:1428] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260619 06:33:40.906347 30720 ts_tablet_manager.cc:1428] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
W20260619 06:33:40.946871 30422 tablet.cc:2369] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:33:40.985081 30561 tablet.cc:2369] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:33:41.005599 30700 tablet.cc:2369] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:33:41.202287 30726 raft_consensus.cc:482] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:33:41.202567 30726 raft_consensus.cc:504] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } }
I20260619 06:33:41.204160 30726 leader_election.cc:288] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20405f9eb2a14ef79a970e920c7b6f98 (127.28.104.193:46647), f61e6ef921274d3aacf8e7d82eb43116 (127.28.104.194:40495)
I20260619 06:33:41.209807 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.204681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:42495 (local address 127.28.104.194:40495)
0619 06:33:41.204869 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:41.204876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:41.205074 (+ 198us) server_negotiation.cc:407] Connection header received
0619 06:33:41.205316 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:41.205326 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:41.205407 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:41.205491 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:41.205862 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.206571 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.208175 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.208908 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.209103 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.209197 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:41.209439 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 06:33:41.209562 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:33:41.209859 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.204583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40495 (local address 127.28.104.195:42495)
0619 06:33:41.204955 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:41.204970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:41.205168 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:41.205509 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:41.205513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:41.205538 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:41.205730 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.205738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.206712 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.206718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.208010 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.208021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.209088 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.209094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.209282 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.209305 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:33:41.209414 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 06:33:41.209553 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":3}
I20260619 06:33:41.210475 30725 raft_consensus.cc:482] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:33:41.210842 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.204565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:35469 (local address 127.28.104.193:46647)
0619 06:33:41.204799 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:41.204805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:41.205413 (+ 608us) server_negotiation.cc:407] Connection header received
0619 06:33:41.205602 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:41.205608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:41.205693 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:41.205794 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:41.206311 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.207027 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.208638 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.209556 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.209730 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.209809 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:41.210437 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 06:33:41.210600 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 06:33:41.210800 30725 raft_consensus.cc:504] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } }
I20260619 06:33:41.210947 30510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385644e8ad3a48f28818a8cf468d2906" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61e6ef921274d3aacf8e7d82eb43116" is_pre_election: true
I20260619 06:33:41.211581 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.204619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:46647 (local address 127.28.104.195:35469)
0619 06:33:41.205304 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:41.205321 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:41.205441 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:41.205808 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:41.205813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:41.205835 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:41.206161 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.206170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.207169 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.207175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.208469 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.208481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.209780 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.209787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.210006 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.210051 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:33:41.211011 (+ 960us) client_negotiation.cc:240] Negotiation successful
0619 06:33:41.211160 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":420,"threads_started":1}
I20260619 06:33:41.211531 30510 raft_consensus.cc:2454] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 0.
I20260619 06:33:41.212320 30583 leader_election.cc:302] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc580958ba31435ea2cfd72c89c5a083, f61e6ef921274d3aacf8e7d82eb43116; no voters:
I20260619 06:33:41.212754 30726 raft_consensus.cc:2781] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:33:41.212617 30372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385644e8ad3a48f28818a8cf468d2906" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20405f9eb2a14ef79a970e920c7b6f98" is_pre_election: true
I20260619 06:33:41.212921 30726 raft_consensus.cc:482] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:33:41.213052 30726 raft_consensus.cc:3037] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:41.213176 30372 raft_consensus.cc:2454] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 0.
I20260619 06:33:41.217360 30726 raft_consensus.cc:504] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } }
I20260619 06:33:41.217957 30725 leader_election.cc:288] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc580958ba31435ea2cfd72c89c5a083 (127.28.104.195:38509), f61e6ef921274d3aacf8e7d82eb43116 (127.28.104.194:40495)
I20260619 06:33:41.218603 30726 leader_election.cc:288] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 election: Requested vote from peers 20405f9eb2a14ef79a970e920c7b6f98 (127.28.104.193:46647), f61e6ef921274d3aacf8e7d82eb43116 (127.28.104.194:40495)
I20260619 06:33:41.218842 30372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385644e8ad3a48f28818a8cf468d2906" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20405f9eb2a14ef79a970e920c7b6f98"
I20260619 06:33:41.219087 30372 raft_consensus.cc:3037] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:41.219233 30510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385644e8ad3a48f28818a8cf468d2906" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61e6ef921274d3aacf8e7d82eb43116"
I20260619 06:33:41.219460 30510 raft_consensus.cc:3037] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:41.222878 30510 raft_consensus.cc:2454] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 1.
I20260619 06:33:41.223047 30372 raft_consensus.cc:2454] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 1.
I20260619 06:33:41.223330 30583 leader_election.cc:302] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc580958ba31435ea2cfd72c89c5a083, f61e6ef921274d3aacf8e7d82eb43116; no voters:
I20260619 06:33:41.223841 30726 raft_consensus.cc:2781] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:41.224363 30726 raft_consensus.cc:686] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 1 LEADER]: Becoming Leader. State: Replica: bc580958ba31435ea2cfd72c89c5a083, State: Running, Role: LEADER
I20260619 06:33:41.225363 30726 consensus_queue.cc:237] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } }
I20260619 06:33:41.225953 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.214256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:60185 (local address 127.28.104.194:40495)
0619 06:33:41.214418 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:41.214427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:41.219106 (+ 4679us) server_negotiation.cc:407] Connection header received
0619 06:33:41.219374 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:41.219381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:41.219466 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:41.219542 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:41.220388 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.221633 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.223952 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.224953 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.225186 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.225268 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:41.225538 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:33:41.225665 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:33:41.226028 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.213822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40495 (local address 127.28.104.193:60185)
0619 06:33:41.218989 (+ 5167us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:41.219007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:41.219211 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:41.219740 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:41.219746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:41.219781 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:41.220184 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.220196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.221835 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.221840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.223804 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.223815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.225211 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.225217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.225378 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.225403 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:33:41.225520 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 06:33:41.225666 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5039,"thread_start_us":4462,"threads_started":1}
I20260619 06:33:41.226707 30510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385644e8ad3a48f28818a8cf468d2906" candidate_uuid: "20405f9eb2a14ef79a970e920c7b6f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61e6ef921274d3aacf8e7d82eb43116" is_pre_election: true
I20260619 06:33:41.227036 30510 raft_consensus.cc:2379] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20405f9eb2a14ef79a970e920c7b6f98 in current term 1: Already voted for candidate bc580958ba31435ea2cfd72c89c5a083 in this term.
I20260619 06:33:41.227241 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.214151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:36615 (local address 127.28.104.195:38509)
0619 06:33:41.214343 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:41.214349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:41.217696 (+ 3347us) server_negotiation.cc:407] Connection header received
0619 06:33:41.217882 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:41.217887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:41.217992 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:41.218082 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:41.219290 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.220356 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.223544 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.224929 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.225090 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.225175 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:41.226921 (+ 1746us) server_negotiation.cc:299] Negotiation successful
0619 06:33:41.227022 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:33:41.227818 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.213697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:38509 (local address 127.28.104.193:36615)
0619 06:33:41.217584 (+ 3887us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:41.217601 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:41.217762 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:41.218310 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:41.218317 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:41.218359 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:41.219105 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.219121 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.220536 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.220541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.223305 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.223323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.226547 (+ 3224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.226553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.226719 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.226758 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:33:41.226925 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 06:33:41.227263 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3753,"thread_start_us":3568,"threads_started":1}
I20260619 06:33:41.228842 30654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385644e8ad3a48f28818a8cf468d2906" candidate_uuid: "20405f9eb2a14ef79a970e920c7b6f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc580958ba31435ea2cfd72c89c5a083" is_pre_election: true
I20260619 06:33:41.230227 30301 leader_election.cc:302] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20405f9eb2a14ef79a970e920c7b6f98; no voters: bc580958ba31435ea2cfd72c89c5a083, f61e6ef921274d3aacf8e7d82eb43116
I20260619 06:33:41.230628 30725 raft_consensus.cc:2726] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 06:33:41.231844 30214 catalog_manager.cc:5495] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc580958ba31435ea2cfd72c89c5a083 (127.28.104.195). New cstate: current_term: 1 leader_uuid: "bc580958ba31435ea2cfd72c89c5a083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } health_report { overall_health: UNKNOWN } } }
I20260619 06:33:41.307963 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.302416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.28.104.195:38509)
0619 06:33:41.302622 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:41.302629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:41.302818 (+ 189us) server_negotiation.cc:407] Connection header received
0619 06:33:41.303113 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:41.303117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:41.303206 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:41.303327 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:41.303605 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.304428 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.305757 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.306179 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.306276 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.306746 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:41.307331 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:41.307447 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:41.307548 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 06:33:41.307618 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:33:41.316056 30510 raft_consensus.cc:1261] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Refusing update from remote peer bc580958ba31435ea2cfd72c89c5a083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:41.316115 30372 raft_consensus.cc:1261] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Refusing update from remote peer bc580958ba31435ea2cfd72c89c5a083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:41.316932 30726 consensus_queue.cc:1035] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:41.317286 30730 consensus_queue.cc:1035] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:41.351605 30213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:41.351903 30213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:41.355419 30213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b: Sending DeleteTablet for 3 replicas of tablet 385644e8ad3a48f28818a8cf468d2906
I20260619 06:33:41.356451 30629 tablet_service.cc:1514] Processing DeleteTablet for tablet 385644e8ad3a48f28818a8cf468d2906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:41 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:55938
I20260619 06:33:41.356616 30351 tablet_service.cc:1514] Processing DeleteTablet for tablet 385644e8ad3a48f28818a8cf468d2906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:41 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:34126
I20260619 06:33:41.357625 30490 tablet_service.cc:1514] Processing DeleteTablet for tablet 385644e8ad3a48f28818a8cf468d2906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:41 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:46214
I20260619 06:33:41.357919 30214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:33:41.358186 30744 tablet_replica.cc:330] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: stopping tablet replica
W20260619 06:33:41.358634 30214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:33:41.358841 30745 tablet_replica.cc:330] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: stopping tablet replica
I20260619 06:33:41.358901 30744 raft_consensus.cc:2229] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:41.359330 30745 raft_consensus.cc:2229] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:41.359395 30744 raft_consensus.cc:2258] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:41.359954 30745 raft_consensus.cc:2258] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:41.361500 30744 ts_tablet_manager.cc:1905] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:41.362459 30745 ts_tablet_manager.cc:1905] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:41.367769 30351 tablet_service.cc:1467] Processing CreateTablet for tablet c2ae36c90c0c4994baad8e01b05e5a02 (DEFAULT_TABLE table=test-table [id=a048d275888747b89a5c4293617550f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:41.368358 30351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ae36c90c0c4994baad8e01b05e5a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:41.368425 30744 ts_tablet_manager.cc:1918] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:33:41.368594 30744 log.cc:1196] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/385644e8ad3a48f28818a8cf468d2906
I20260619 06:33:41.369200 30744 ts_tablet_manager.cc:1939] T 385644e8ad3a48f28818a8cf468d2906 P 20405f9eb2a14ef79a970e920c7b6f98: Deleting consensus metadata
I20260619 06:33:41.369830 30745 ts_tablet_manager.cc:1918] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:33:41.370043 30745 log.cc:1196] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/385644e8ad3a48f28818a8cf468d2906
I20260619 06:33:41.370613 30745 ts_tablet_manager.cc:1939] T 385644e8ad3a48f28818a8cf468d2906 P f61e6ef921274d3aacf8e7d82eb43116: Deleting consensus metadata
I20260619 06:33:41.371266 30198 catalog_manager.cc:4841] TS 20405f9eb2a14ef79a970e920c7b6f98 (127.28.104.193:46647): tablet 385644e8ad3a48f28818a8cf468d2906 (table test-table [id=92b271012fb7490e8e2570648964482e]) successfully deleted
I20260619 06:33:41.373310 30201 catalog_manager.cc:4841] TS f61e6ef921274d3aacf8e7d82eb43116 (127.28.104.194:40495): tablet 385644e8ad3a48f28818a8cf468d2906 (table test-table [id=92b271012fb7490e8e2570648964482e]) successfully deleted
I20260619 06:33:41.373966 30490 tablet_service.cc:1467] Processing CreateTablet for tablet c2ae36c90c0c4994baad8e01b05e5a02 (DEFAULT_TABLE table=test-table [id=a048d275888747b89a5c4293617550f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:41.373904 30628 tablet_service.cc:1467] Processing CreateTablet for tablet c2ae36c90c0c4994baad8e01b05e5a02 (DEFAULT_TABLE table=test-table [id=a048d275888747b89a5c4293617550f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:41.374392 30490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ae36c90c0c4994baad8e01b05e5a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:41.374660 30628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ae36c90c0c4994baad8e01b05e5a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:41.384116 30719 tablet_bootstrap.cc:492] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083: Bootstrap starting.
I20260619 06:33:41.385215 30718 tablet_bootstrap.cc:492] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116: Bootstrap starting.
I20260619 06:33:41.385324 30720 tablet_bootstrap.cc:492] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98: Bootstrap starting.
I20260619 06:33:41.385562 30743 tablet_replica.cc:330] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: stopping tablet replica
I20260619 06:33:41.386016 30743 raft_consensus.cc:2229] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:33:41.386704 30743 raft_consensus.cc:2258] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:41.386855 30718 tablet_bootstrap.cc:654] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:41.386917 30720 tablet_bootstrap.cc:654] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:41.388839 30719 tablet_bootstrap.cc:654] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:41.396451 30719 tablet_bootstrap.cc:492] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083: No bootstrap required, opened a new log
I20260619 06:33:41.396517 30720 tablet_bootstrap.cc:492] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98: No bootstrap required, opened a new log
I20260619 06:33:41.396615 30719 ts_tablet_manager.cc:1397] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260619 06:33:41.396682 30720 ts_tablet_manager.cc:1397] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20260619 06:33:41.397238 30743 ts_tablet_manager.cc:1905] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:41.397509 30719 raft_consensus.cc:348] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.397533 30720 raft_consensus.cc:348] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.397742 30719 raft_consensus.cc:374] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:41.397775 30720 raft_consensus.cc:374] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:41.397876 30720 raft_consensus.cc:729] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20405f9eb2a14ef79a970e920c7b6f98, State: Initialized, Role: FOLLOWER
I20260619 06:33:41.397876 30719 raft_consensus.cc:729] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc580958ba31435ea2cfd72c89c5a083, State: Initialized, Role: FOLLOWER
I20260619 06:33:41.398175 30719 consensus_queue.cc:260] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.398188 30720 consensus_queue.cc:260] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.398779 30719 ts_tablet_manager.cc:1428] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 06:33:41.401994 30720 ts_tablet_manager.cc:1428] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:33:41.402969 30718 tablet_bootstrap.cc:492] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116: No bootstrap required, opened a new log
I20260619 06:33:41.403134 30718 ts_tablet_manager.cc:1397] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20260619 06:33:41.403965 30718 raft_consensus.cc:348] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.404202 30718 raft_consensus.cc:374] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:41.404421 30718 raft_consensus.cc:729] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61e6ef921274d3aacf8e7d82eb43116, State: Initialized, Role: FOLLOWER
I20260619 06:33:41.404825 30743 ts_tablet_manager.cc:1918] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:33:41.404702 30718 consensus_queue.cc:260] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.405011 30743 log.cc:1196] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/385644e8ad3a48f28818a8cf468d2906
I20260619 06:33:41.405337 30718 ts_tablet_manager.cc:1428] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 06:33:41.405684 30743 ts_tablet_manager.cc:1939] T 385644e8ad3a48f28818a8cf468d2906 P bc580958ba31435ea2cfd72c89c5a083: Deleting consensus metadata
I20260619 06:33:41.407420 30198 catalog_manager.cc:4841] TS bc580958ba31435ea2cfd72c89c5a083 (127.28.104.195:38509): tablet 385644e8ad3a48f28818a8cf468d2906 (table test-table [id=92b271012fb7490e8e2570648964482e]) successfully deleted
I20260619 06:33:41.566967 30726 raft_consensus.cc:482] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:33:41.567222 30726 raft_consensus.cc:504] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.567902 30726 leader_election.cc:288] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20405f9eb2a14ef79a970e920c7b6f98 (127.28.104.193:46647), f61e6ef921274d3aacf8e7d82eb43116 (127.28.104.194:40495)
I20260619 06:33:41.568532 30372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ae36c90c0c4994baad8e01b05e5a02" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20405f9eb2a14ef79a970e920c7b6f98" is_pre_election: true
I20260619 06:33:41.568589 30510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ae36c90c0c4994baad8e01b05e5a02" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61e6ef921274d3aacf8e7d82eb43116" is_pre_election: true
I20260619 06:33:41.568794 30372 raft_consensus.cc:2454] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 0.
I20260619 06:33:41.568892 30510 raft_consensus.cc:2454] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 0.
I20260619 06:33:41.569352 30580 leader_election.cc:302] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20405f9eb2a14ef79a970e920c7b6f98, bc580958ba31435ea2cfd72c89c5a083; no voters:
I20260619 06:33:41.569762 30726 raft_consensus.cc:2781] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:33:41.569887 30726 raft_consensus.cc:482] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:33:41.569989 30726 raft_consensus.cc:3037] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:41.572175 30726 raft_consensus.cc:504] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.572885 30726 leader_election.cc:288] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 election: Requested vote from peers 20405f9eb2a14ef79a970e920c7b6f98 (127.28.104.193:46647), f61e6ef921274d3aacf8e7d82eb43116 (127.28.104.194:40495)
I20260619 06:33:41.573465 30372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ae36c90c0c4994baad8e01b05e5a02" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20405f9eb2a14ef79a970e920c7b6f98"
I20260619 06:33:41.573477 30510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ae36c90c0c4994baad8e01b05e5a02" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61e6ef921274d3aacf8e7d82eb43116"
I20260619 06:33:41.573691 30372 raft_consensus.cc:3037] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:41.573709 30510 raft_consensus.cc:3037] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:41.575922 30372 raft_consensus.cc:2454] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 1.
I20260619 06:33:41.576364 30580 leader_election.cc:302] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20405f9eb2a14ef79a970e920c7b6f98, bc580958ba31435ea2cfd72c89c5a083; no voters:
I20260619 06:33:41.576673 30726 raft_consensus.cc:2781] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:41.576761 30510 raft_consensus.cc:2454] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 1.
I20260619 06:33:41.576845 30726 raft_consensus.cc:686] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 1 LEADER]: Becoming Leader. State: Replica: bc580958ba31435ea2cfd72c89c5a083, State: Running, Role: LEADER
I20260619 06:33:41.577173 30726 consensus_queue.cc:237] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.580207 30212 catalog_manager.cc:5495] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc580958ba31435ea2cfd72c89c5a083 (127.28.104.195). New cstate: current_term: 1 leader_uuid: "bc580958ba31435ea2cfd72c89c5a083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } health_report { overall_health: HEALTHY } } }
Jun 19 06:33:41 dist-test-slave-nf4n krb5kdc[30169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850819, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:41.668031 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.647538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.28.104.254:35503)
0619 06:33:41.648252 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:41.648258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:41.648545 (+ 287us) server_negotiation.cc:407] Connection header received
0619 06:33:41.648755 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:41.648759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:41.648815 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:41.648906 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:41.650474 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.651259 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.653180 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.653711 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.653806 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.662264 (+ 8458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:41.662270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:41.662273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:41.662306 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:41.664999 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:41.666009 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:41.666014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:41.666016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:41.666178 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:41.666658 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:41.666661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:41.666663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:41.666938 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:41.667047 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:41.667580 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 06:33:41.667751 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":333,"threads_started":1}
I20260619 06:33:41.668423 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.647516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35503 (local address 127.0.0.1:55252)
0619 06:33:41.648422 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:41.648443 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:41.648613 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:41.648992 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:41.648998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:41.649993 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:41.650299 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.650313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.651418 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.651425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.653010 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.653058 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.653887 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.653894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.654099 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.658783 (+ 4684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:41.658811 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:41.662024 (+ 3213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:41.665205 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:41.665221 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:41.665232 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:41.665844 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:41.666307 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:41.666312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:41.666316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:41.666539 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:41.667073 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:41.667083 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:41.667291 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 06:33:41.667481 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 06:33:41.667946 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":118,"thread_start_us":614,"threads_started":1}
I20260619 06:33:41.674304 30212 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:55252:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:33:41.675593 30212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:33:41.683349 30351 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff4c311812f4905819f2eb6d3d874a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f2cd1fa43d544bd8ca33b83593b98ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:33:41.684039 30351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff4c311812f4905819f2eb6d3d874a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:41.689464 30490 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff4c311812f4905819f2eb6d3d874a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f2cd1fa43d544bd8ca33b83593b98ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:33:41.690126 30490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff4c311812f4905819f2eb6d3d874a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:41.690143 30720 tablet_bootstrap.cc:492] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: Bootstrap starting.
I20260619 06:33:41.691474 30628 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff4c311812f4905819f2eb6d3d874a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f2cd1fa43d544bd8ca33b83593b98ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:33:41.691944 30720 tablet_bootstrap.cc:654] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:41.692116 30628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff4c311812f4905819f2eb6d3d874a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:41.700201 30719 tablet_bootstrap.cc:492] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: Bootstrap starting.
I20260619 06:33:41.702262 30719 tablet_bootstrap.cc:654] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:41.707402 30718 tablet_bootstrap.cc:492] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: Bootstrap starting.
I20260619 06:33:41.710908 30718 tablet_bootstrap.cc:654] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:41.711737 30719 tablet_bootstrap.cc:492] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: No bootstrap required, opened a new log
I20260619 06:33:41.712133 30719 ts_tablet_manager.cc:1397] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260619 06:33:41.713397 30719 raft_consensus.cc:348] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.713814 30719 raft_consensus.cc:374] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:41.713941 30719 raft_consensus.cc:729] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc580958ba31435ea2cfd72c89c5a083, State: Initialized, Role: FOLLOWER
I20260619 06:33:41.714316 30719 consensus_queue.cc:260] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.714881 30726 tablet_replica.cc:439] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } }
I20260619 06:33:41.715207 30726 tablet_replica.cc:442] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:41.715766 30719 ts_tablet_manager.cc:1428] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:33:41.717684 30720 tablet_bootstrap.cc:492] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: No bootstrap required, opened a new log
I20260619 06:33:41.718791 30720 ts_tablet_manager.cc:1397] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.010s
I20260619 06:33:41.719914 30720 raft_consensus.cc:348] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.721294 30720 raft_consensus.cc:374] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:41.721405 30720 raft_consensus.cc:729] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20405f9eb2a14ef79a970e920c7b6f98, State: Initialized, Role: FOLLOWER
I20260619 06:33:41.721697 30720 consensus_queue.cc:260] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.722363 30720 ts_tablet_manager.cc:1428] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260619 06:33:41.722292 30725 tablet_replica.cc:439] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } }
I20260619 06:33:41.722628 30725 tablet_replica.cc:442] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:41.724954 30718 tablet_bootstrap.cc:492] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: No bootstrap required, opened a new log
I20260619 06:33:41.725162 30718 ts_tablet_manager.cc:1397] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.002s
I20260619 06:33:41.725953 30718 raft_consensus.cc:348] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.726567 30718 raft_consensus.cc:374] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:41.726733 30718 raft_consensus.cc:729] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61e6ef921274d3aacf8e7d82eb43116, State: Initialized, Role: FOLLOWER
I20260619 06:33:41.727303 30718 consensus_queue.cc:260] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.728138 30724 tablet_replica.cc:439] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } }
I20260619 06:33:41.728477 30724 tablet_replica.cc:442] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:41.728931 30718 ts_tablet_manager.cc:1428] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:33:41.868342 30726 raft_consensus.cc:482] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:33:41.868646 30726 raft_consensus.cc:504] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.869434 30726 leader_election.cc:288] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20405f9eb2a14ef79a970e920c7b6f98 (127.28.104.193:46647), f61e6ef921274d3aacf8e7d82eb43116 (127.28.104.194:40495)
I20260619 06:33:41.870007 30510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff4c311812f4905819f2eb6d3d874a1" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61e6ef921274d3aacf8e7d82eb43116" is_pre_election: true
I20260619 06:33:41.870026 30372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff4c311812f4905819f2eb6d3d874a1" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20405f9eb2a14ef79a970e920c7b6f98" is_pre_election: true
I20260619 06:33:41.870296 30510 raft_consensus.cc:2454] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 0.
I20260619 06:33:41.870308 30372 raft_consensus.cc:2454] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 0.
I20260619 06:33:41.870754 30583 leader_election.cc:302] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc580958ba31435ea2cfd72c89c5a083, f61e6ef921274d3aacf8e7d82eb43116; no voters:
I20260619 06:33:41.871047 30726 raft_consensus.cc:2781] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:33:41.871157 30726 raft_consensus.cc:482] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:33:41.871241 30726 raft_consensus.cc:3037] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:41.873358 30726 raft_consensus.cc:504] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.873859 30726 leader_election.cc:288] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 election: Requested vote from peers 20405f9eb2a14ef79a970e920c7b6f98 (127.28.104.193:46647), f61e6ef921274d3aacf8e7d82eb43116 (127.28.104.194:40495)
I20260619 06:33:41.874310 30510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff4c311812f4905819f2eb6d3d874a1" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61e6ef921274d3aacf8e7d82eb43116"
I20260619 06:33:41.874307 30372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff4c311812f4905819f2eb6d3d874a1" candidate_uuid: "bc580958ba31435ea2cfd72c89c5a083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20405f9eb2a14ef79a970e920c7b6f98"
I20260619 06:33:41.874511 30510 raft_consensus.cc:3037] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:41.874539 30372 raft_consensus.cc:3037] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:41.876358 30510 raft_consensus.cc:2454] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 1.
I20260619 06:33:41.876441 30372 raft_consensus.cc:2454] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc580958ba31435ea2cfd72c89c5a083 in term 1.
I20260619 06:33:41.876825 30583 leader_election.cc:302] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc580958ba31435ea2cfd72c89c5a083, f61e6ef921274d3aacf8e7d82eb43116; no voters:
I20260619 06:33:41.877162 30726 raft_consensus.cc:2781] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:41.877346 30726 raft_consensus.cc:686] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [term 1 LEADER]: Becoming Leader. State: Replica: bc580958ba31435ea2cfd72c89c5a083, State: Running, Role: LEADER
I20260619 06:33:41.877633 30726 consensus_queue.cc:237] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } }
I20260619 06:33:41.879156 30730 tablet_replica.cc:439] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: TxnStatusTablet state changed. Reason: New leader bc580958ba31435ea2cfd72c89c5a083. Latest consensus state: current_term: 1 leader_uuid: "bc580958ba31435ea2cfd72c89c5a083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } }
I20260619 06:33:41.879387 30730 tablet_replica.cc:442] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:33:41.879946 30761 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:33:41.880235 30761 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:33:41.881512 30212 catalog_manager.cc:5495] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc580958ba31435ea2cfd72c89c5a083 (127.28.104.195). New cstate: current_term: 1 leader_uuid: "bc580958ba31435ea2cfd72c89c5a083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } health_report { overall_health: HEALTHY } } }
I20260619 06:33:41.938882 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.933328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:38509 (local address 127.0.0.1:55950)
0619 06:33:41.933550 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:41.933570 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:41.933728 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:41.934320 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:41.934328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:41.934342 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:33:41.934533 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.934543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.935697 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.935705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.936693 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.936704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.937531 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.937537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.937773 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.937832 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:33:41.938258 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:33:41.938284 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:33:41.938381 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:33:41.938561 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 06:33:41.938920 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.933281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.28.104.195:38509)
0619 06:33:41.934019 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:41.934023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:41.934037 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:33:41.934094 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:41.934097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:41.934165 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:41.934270 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:41.934731 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.935545 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.936839 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.937370 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.937516 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.938039 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:41.938143 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:41.938243 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:41.938431 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 06:33:41.938508 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":447,"threads_started":1}
I20260619 06:33:41.943483 30510 raft_consensus.cc:1261] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Refusing update from remote peer bc580958ba31435ea2cfd72c89c5a083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:41.943526 30372 raft_consensus.cc:1261] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Refusing update from remote peer bc580958ba31435ea2cfd72c89c5a083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:41.944125 30726 consensus_queue.cc:1035] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:41.944425 30730 consensus_queue.cc:1035] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:41.955052 30724 tablet_replica.cc:439] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: TxnStatusTablet state changed. Reason: New leader bc580958ba31435ea2cfd72c89c5a083. Latest consensus state: current_term: 1 leader_uuid: "bc580958ba31435ea2cfd72c89c5a083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } }
I20260619 06:33:41.955349 30724 tablet_replica.cc:442] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:41.959805 30725 tablet_replica.cc:439] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: TxnStatusTablet state changed. Reason: New leader bc580958ba31435ea2cfd72c89c5a083. Latest consensus state: current_term: 1 leader_uuid: "bc580958ba31435ea2cfd72c89c5a083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } }
I20260619 06:33:41.960079 30725 tablet_replica.cc:442] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:41.961404 30764 mvcc.cc:205] Tried to move back new op lower bound from 7298460966673309696 to 7298460966413410304. Current Snapshot: MvccSnapshot[applied={T|T < 7298460966673309696}]
I20260619 06:33:41.963115 30726 tablet_replica.cc:439] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc580958ba31435ea2cfd72c89c5a083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } }
I20260619 06:33:41.963501 30726 tablet_replica.cc:442] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:33:41.973696 30725 tablet_replica.cc:439] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc580958ba31435ea2cfd72c89c5a083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } }
I20260619 06:33:41.974073 30725 tablet_replica.cc:442] T 1ff4c311812f4905819f2eb6d3d874a1 P 20405f9eb2a14ef79a970e920c7b6f98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:41.977007 30726 tablet_replica.cc:439] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc580958ba31435ea2cfd72c89c5a083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } }
I20260619 06:33:41.977332 30726 tablet_replica.cc:442] T 1ff4c311812f4905819f2eb6d3d874a1 P bc580958ba31435ea2cfd72c89c5a083: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:33:41.984017 30724 tablet_replica.cc:439] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc580958ba31435ea2cfd72c89c5a083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 } } peers { permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 } } peers { permanent_uuid: "bc580958ba31435ea2cfd72c89c5a083" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38509 } } }
I20260619 06:33:41.984316 30724 tablet_replica.cc:442] T 1ff4c311812f4905819f2eb6d3d874a1 P f61e6ef921274d3aacf8e7d82eb43116: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:41.994249 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.988373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:37589 (local address 127.28.104.195:38509)
0619 06:33:41.988530 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:41.988534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:41.989283 (+ 749us) server_negotiation.cc:407] Connection header received
0619 06:33:41.989491 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:41.989496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:41.989567 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:41.989662 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:41.990119 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.990851 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.992438 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.992900 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.993063 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.993522 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:41.993621 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:41.993720 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:41.993864 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 06:33:41.993932 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:33:41.994467 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:41.988516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:38509 (local address 127.28.104.195:37589)
0619 06:33:41.989191 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:41.989207 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:41.989376 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:41.989676 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:41.989680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:41.989689 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:33:41.989985 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.989993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.990980 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.990985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.992299 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:41.992312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:41.993136 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:41.993150 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:41.993268 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:41.993326 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:33:41.993719 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:33:41.993730 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:33:41.993845 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 06:33:41.993974 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":331,"threads_started":1}
I20260619 06:33:42.003597 30510 raft_consensus.cc:1261] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Refusing update from remote peer bc580958ba31435ea2cfd72c89c5a083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:42.003746 30372 raft_consensus.cc:1261] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Refusing update from remote peer bc580958ba31435ea2cfd72c89c5a083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:42.004244 30730 consensus_queue.cc:1035] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61e6ef921274d3aacf8e7d82eb43116" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:42.004503 30726 consensus_queue.cc:1035] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20405f9eb2a14ef79a970e920c7b6f98" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:42.069216 30214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:42.069423 30214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:42.072578 30214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0dc9a3c15c9473ca697c802b4ea0d6b: Sending DeleteTablet for 3 replicas of tablet c2ae36c90c0c4994baad8e01b05e5a02
I20260619 06:33:42.073483 30351 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ae36c90c0c4994baad8e01b05e5a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:42 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:34126
I20260619 06:33:42.073599 30490 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ae36c90c0c4994baad8e01b05e5a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:42 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:46214
I20260619 06:33:42.073807 30628 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ae36c90c0c4994baad8e01b05e5a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:42 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:55938
I20260619 06:33:42.084735 30780 tablet_replica.cc:330] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116: stopping tablet replica
I20260619 06:33:42.085179 30780 raft_consensus.cc:2229] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:42.085577 30780 raft_consensus.cc:2258] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:42.086930 30780 ts_tablet_manager.cc:1905] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:42.087550 30779 tablet_replica.cc:330] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98: stopping tablet replica
I20260619 06:33:42.087843 30779 raft_consensus.cc:2229] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:42.089263 30779 raft_consensus.cc:2258] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:42.092068 30779 ts_tablet_manager.cc:1905] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:42.103267 30780 ts_tablet_manager.cc:1918] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:33:42.103597 30781 tablet_replica.cc:330] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083: stopping tablet replica
I20260619 06:33:42.103902 30781 raft_consensus.cc:2229] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:33:42.104524 30781 raft_consensus.cc:2258] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:42.104790 30780 log.cc:1196] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/c2ae36c90c0c4994baad8e01b05e5a02
I20260619 06:33:42.105410 30780 ts_tablet_manager.cc:1939] T c2ae36c90c0c4994baad8e01b05e5a02 P f61e6ef921274d3aacf8e7d82eb43116: Deleting consensus metadata
I20260619 06:33:42.106632 30779 ts_tablet_manager.cc:1918] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:33:42.106799 30781 ts_tablet_manager.cc:1905] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:42.108089 30201 catalog_manager.cc:4841] TS f61e6ef921274d3aacf8e7d82eb43116 (127.28.104.194:40495): tablet c2ae36c90c0c4994baad8e01b05e5a02 (table test-table [id=a048d275888747b89a5c4293617550f6]) successfully deleted
I20260619 06:33:42.108464 30779 log.cc:1196] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/c2ae36c90c0c4994baad8e01b05e5a02
I20260619 06:33:42.109125 30779 ts_tablet_manager.cc:1939] T c2ae36c90c0c4994baad8e01b05e5a02 P 20405f9eb2a14ef79a970e920c7b6f98: Deleting consensus metadata
I20260619 06:33:42.110483 30198 catalog_manager.cc:4841] TS 20405f9eb2a14ef79a970e920c7b6f98 (127.28.104.193:46647): tablet c2ae36c90c0c4994baad8e01b05e5a02 (table test-table [id=a048d275888747b89a5c4293617550f6]) successfully deleted
I20260619 06:33:42.115185 30781 ts_tablet_manager.cc:1918] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:33:42.115325 30781 log.cc:1196] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/c2ae36c90c0c4994baad8e01b05e5a02
I20260619 06:33:42.115760 30781 ts_tablet_manager.cc:1939] T c2ae36c90c0c4994baad8e01b05e5a02 P bc580958ba31435ea2cfd72c89c5a083: Deleting consensus metadata
I20260619 06:33:42.116781 30198 catalog_manager.cc:4841] TS bc580958ba31435ea2cfd72c89c5a083 (127.28.104.195:38509): tablet c2ae36c90c0c4994baad8e01b05e5a02 (table test-table [id=a048d275888747b89a5c4293617550f6]) successfully deleted
Jun 19 06:33:42 dist-test-slave-nf4n krb5kdc[30169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:33:42.129889 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:42.085408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.28.104.193:46647)
0619 06:33:42.105328 (+ 19920us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:42.105339 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:42.109692 (+ 4353us) server_negotiation.cc:407] Connection header received
0619 06:33:42.110017 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:42.110023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:42.110109 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:42.110201 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:42.112344 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:42.113712 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:42.115884 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:42.116602 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:42.116734 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:42.122427 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:42.122448 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:42.122472 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:42.122521 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:42.126632 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:42.127358 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:42.127369 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:42.127373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:42.127554 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:42.127995 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:42.128009 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:42.128012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:42.128488 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:42.128921 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:42.129186 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 06:33:42.129454 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19709,"thread_start_us":20035,"threads_started":1}
W20260619 06:33:42.130661 30416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34128
I20260619 06:33:42.148382 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:42.133661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.28.104.254:35503)
0619 06:33:42.133910 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:42.133919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:42.134344 (+ 425us) server_negotiation.cc:407] Connection header received
0619 06:33:42.134451 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:42.134457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:42.134561 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:42.134676 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:42.135822 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:42.137013 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:42.138911 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:42.139771 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:42.139900 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:42.142841 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:42.142847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:42.142852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:42.142894 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:42.145783 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:42.146425 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:42.146431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:42.146435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:42.146566 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:42.146940 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:42.146947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:42.146950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:42.147239 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:42.147492 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:42.147827 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 06:33:42.148062 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 06:33:42.149102 30214 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55254
I20260619 06:33:42.151002 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 30285
I20260619 06:33:42.171030 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:42.151832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.28.104.193:46647)
0619 06:33:42.152045 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:42.152055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:42.157550 (+ 5495us) server_negotiation.cc:407] Connection header received
0619 06:33:42.157702 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:42.157709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:42.157803 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:42.157933 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:42.159247 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:42.160395 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:42.161736 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:42.162488 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:42.162633 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:42.164810 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:42.164819 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:42.164824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:42.164870 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:42.168225 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:42.168875 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:42.168881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:42.168885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:42.169125 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:42.169477 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:42.169482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:42.169486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:42.169823 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:42.170088 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:42.170324 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 06:33:42.170554 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 06:33:42.171639 30416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34142
W20260619 06:33:42.172230 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 30285: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:33:42.172431 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 30285
I20260619 06:33:42.190658 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 30425
Jun 19 06:33:42 dist-test-slave-nf4n krb5kdc[30169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:33:42.206692 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:42.191570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.28.104.194:40495)
0619 06:33:42.192318 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:42.192323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:42.192338 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:33:42.192391 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:42.192395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:42.192451 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:42.192578 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:42.193907 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:42.194679 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:42.195915 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:42.196343 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:42.196479 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:42.200544 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:42.200558 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:42.200575 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:42.200611 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:42.203750 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:42.204433 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:42.204442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:42.204445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:42.204581 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:42.204971 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:42.204978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:42.204981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:42.205456 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:42.205770 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:42.206072 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 06:33:42.206326 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":478,"threads_started":1}
W20260619 06:33:42.207394 30555 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226
W20260619 06:33:42.208112 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 30425: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:33:42.208266 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 30425
I20260619 06:33:42.232921 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 30564
Jun 19 06:33:42 dist-test-slave-nf4n krb5kdc[30169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:33:42.249697 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:42.233817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.28.104.195:38509)
0619 06:33:42.234022 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:42.234027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:42.234109 (+ 82us) server_negotiation.cc:407] Connection header received
0619 06:33:42.234361 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:42.234367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:42.234438 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:42.234543 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:42.235950 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:42.236741 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:42.238203 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:42.238765 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:42.238906 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:42.243216 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:42.243231 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:42.243251 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:42.243291 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:42.246686 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:42.247446 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:42.247454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:42.247456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:42.247611 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:42.248018 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:42.248025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:42.248028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:42.248414 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:42.248756 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:42.249115 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 06:33:42.249348 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 06:33:42.250504 30694 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55954
W20260619 06:33:42.251217 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 30564: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:33:42.251397 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 30564
I20260619 06:33:42.275077 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 30182
I20260619 06:33:42.276177 30254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:42.392134 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 30182
2026-06-19T06:33:42Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5022 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:33:42 dist-test-slave-nf4n krb5kdc[30792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:33:42 dist-test-slave-nf4n krb5kdc[30792](info): set up 2 sockets
Jun 19 06:33:42 dist-test-slave-nf4n krb5kdc[30792](info): commencing operation
krb5kdc: starting...
W20260619 06:33:43.484990 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:33:43 dist-test-slave-nf4n krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:33:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:33:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:33:43.675946 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:32843
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:32843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:43.997756 30805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:43.998142 30805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:43.998312 30805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:44.005605 30805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:44.005755 30805 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:33:44.005856 30805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:44.005975 30805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:44.006078 30805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:44.006165 30805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:44.016188 30805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:32843
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:32843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:44.017539 30805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:44.018887 30805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:44.030821 30810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:44.030941 30813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:44.030952 30811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:44.031562 30805 server_base.cc:1034] running on GCE node
I20260619 06:33:44.032330 30805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:44.033841 30805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:44.035122 30805 hybrid_clock.cc:648] HybridClock initialized: now 1781850824035005 us; error 29 us; skew 500 ppm
I20260619 06:33:44.035481 30805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:44 dist-test-slave-nf4n krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850824, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:44.038724 30805 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:33:44.040249 30805 webserver.cc:460] Webserver started at http://127.28.104.254:41833/ using document root <none> and password file <none>
I20260619 06:33:44.040959 30805 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:44.041116 30805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:44.041464 30805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:44.044468 30805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "44de544ccc2f4adaa4bf98a77a72f3f5"
format_stamp: "Formatted at 2026-06-19 06:33:44 on dist-test-slave-nf4n"
server_key: "99245c936cae17a8cc6ca5695039da09"
server_key_iv: "0b5520633d207d7a3c15acaa6be39108"
server_key_version: "encryptionkey@0"
I20260619 06:33:44.045488 30805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "44de544ccc2f4adaa4bf98a77a72f3f5"
format_stamp: "Formatted at 2026-06-19 06:33:44 on dist-test-slave-nf4n"
server_key: "99245c936cae17a8cc6ca5695039da09"
server_key_iv: "0b5520633d207d7a3c15acaa6be39108"
server_key_version: "encryptionkey@0"
I20260619 06:33:44.050518 30805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 06:33:44.053812 30820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:44.055119 30805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:33:44.055302 30805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "44de544ccc2f4adaa4bf98a77a72f3f5"
format_stamp: "Formatted at 2026-06-19 06:33:44 on dist-test-slave-nf4n"
server_key: "99245c936cae17a8cc6ca5695039da09"
server_key_iv: "0b5520633d207d7a3c15acaa6be39108"
server_key_version: "encryptionkey@0"
I20260619 06:33:44.055516 30805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:44.067094 30805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:44.070947 30805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:44.071231 30805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:44.092842 30805 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:32843
I20260619 06:33:44.092864 30881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:32843 every 8 connection(s)
I20260619 06:33:44.094322 30805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:33:44.098250 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:44.104367 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 30805
I20260619 06:33:44.104640 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:33:44.105633 29091 external_mini_cluster.cc:1428] Setting key b30e76b946843d82e6468f437a13f023
I20260619 06:33:44.110232 30882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5: Bootstrap starting.
I20260619 06:33:44.114570 30882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:44.115911 30882 log.cc:824] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:44.119319 30882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5: No bootstrap required, opened a new log
Jun 19 06:33:44 dist-test-slave-nf4n krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:44.127803 30882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44de544ccc2f4adaa4bf98a77a72f3f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32843 } }
I20260619 06:33:44.128307 30882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:44.128448 30882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44de544ccc2f4adaa4bf98a77a72f3f5, State: Initialized, Role: FOLLOWER
I20260619 06:33:44.129226 30882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44de544ccc2f4adaa4bf98a77a72f3f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32843 } }
I20260619 06:33:44.129563 30882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:33:44.129733 30882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:33:44.129923 30882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:44.130905 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:44.107918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.28.104.254:32843)
0619 06:33:44.108792 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:44.108805 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:44.108846 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:33:44.110693 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:44.110717 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:44.111207 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:44.111600 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:44.113655 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:44.115083 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:44.117007 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:44.117851 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:44.117992 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:44.122749 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:44.122758 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:44.122777 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:44.122821 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:44.126078 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:44.126779 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:44.126787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:44.126793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:44.126961 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:44.127508 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:44.127539 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:44.127541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:44.128170 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:44.128561 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:44.129076 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 06:33:44.129605 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":391,"threads_started":1}
I20260619 06:33:44.132838 30882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44de544ccc2f4adaa4bf98a77a72f3f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32843 } }
I20260619 06:33:44.133404 30882 leader_election.cc:302] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44de544ccc2f4adaa4bf98a77a72f3f5; no voters:
I20260619 06:33:44.134068 30882 leader_election.cc:288] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:33:44.134207 30887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:44.134938 30887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [term 1 LEADER]: Becoming Leader. State: Replica: 44de544ccc2f4adaa4bf98a77a72f3f5, State: Running, Role: LEADER
I20260619 06:33:44.135443 30887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44de544ccc2f4adaa4bf98a77a72f3f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32843 } }
I20260619 06:33:44.136150 30882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:44.138401 30889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44de544ccc2f4adaa4bf98a77a72f3f5. Latest consensus state: current_term: 1 leader_uuid: "44de544ccc2f4adaa4bf98a77a72f3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44de544ccc2f4adaa4bf98a77a72f3f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32843 } } }
I20260619 06:33:44.138823 30889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:44.139205 30888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44de544ccc2f4adaa4bf98a77a72f3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44de544ccc2f4adaa4bf98a77a72f3f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32843 } } }
I20260619 06:33:44.139485 30888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:44.141849 30896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:33:44.147564 30896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:33:44.157121 30896 catalog_manager.cc:1347] Generated new cluster ID: 604231d533484380b95981656b01eed2
I20260619 06:33:44.157263 30896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:33:44.177765 30896 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:33:44.179104 30896 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:33:44.192232 30896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5: Generated new TSK 0
I20260619 06:33:44.193192 30896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:33:44.239969 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:32843
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:44.564780 30908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:44.565147 30908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:44.565354 30908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:44.572885 30908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:44.573009 30908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:44.573210 30908 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:33:44.573274 30908 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:33:44.573378 30908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:33:44.583585 30908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.104.254:32843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:44.584941 30908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:44.586406 30908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:44.598301 30913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:44.599592 30914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:44.600307 30916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:44.600755 30908 server_base.cc:1034] running on GCE node
I20260619 06:33:44.601467 30908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:44.602442 30908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:44.603667 30908 hybrid_clock.cc:648] HybridClock initialized: now 1781850824603609 us; error 52 us; skew 500 ppm
I20260619 06:33:44.603979 30908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:44 dist-test-slave-nf4n krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850824, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:44.607079 30908 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:33:44.608511 30908 webserver.cc:460] Webserver started at http://127.28.104.193:45651/ using document root <none> and password file <none>
I20260619 06:33:44.609293 30908 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:44.609437 30908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:44.609747 30908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:44.612669 30908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "fd77a969ff064b849ebd584641a116f1"
format_stamp: "Formatted at 2026-06-19 06:33:44 on dist-test-slave-nf4n"
server_key: "1da4e98f3d71819326b2f8ba0d4fd8d1"
server_key_iv: "a085c4e6546d1cc6cd76b4de44233467"
server_key_version: "encryptionkey@0"
I20260619 06:33:44.613476 30908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "fd77a969ff064b849ebd584641a116f1"
format_stamp: "Formatted at 2026-06-19 06:33:44 on dist-test-slave-nf4n"
server_key: "1da4e98f3d71819326b2f8ba0d4fd8d1"
server_key_iv: "a085c4e6546d1cc6cd76b4de44233467"
server_key_version: "encryptionkey@0"
I20260619 06:33:44.618808 30908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 06:33:44.622268 30923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:44.623557 30908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:33:44.623725 30908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "fd77a969ff064b849ebd584641a116f1"
format_stamp: "Formatted at 2026-06-19 06:33:44 on dist-test-slave-nf4n"
server_key: "1da4e98f3d71819326b2f8ba0d4fd8d1"
server_key_iv: "a085c4e6546d1cc6cd76b4de44233467"
server_key_version: "encryptionkey@0"
I20260619 06:33:44.623924 30908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:44.645292 30908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:44.649546 30908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:44.649837 30908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:44.652513 30908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:44.652659 30908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:44.652801 30908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:44.652930 30908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:33:44 dist-test-slave-nf4n krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850824, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:44.676800 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:44.653168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:58541 (local address 127.28.104.254:32843)
0619 06:33:44.653812 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:44.653819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:44.654906 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 06:33:44.656997 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:44.657003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:44.657125 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:44.657276 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:44.659918 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:44.661385 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:44.664067 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:44.664946 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:44.665144 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:44.669485 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:44.669494 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:44.669502 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:44.669560 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:44.673224 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:44.674363 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:44.674371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:44.674378 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:44.674516 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:44.674975 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:44.674981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:44.674984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:44.675295 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:44.675449 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:44.676205 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 06:33:44.676431 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":389,"threads_started":1}
I20260619 06:33:44.678267 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:44.653841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:32843 (local address 127.28.104.193:58541)
0619 06:33:44.654713 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:44.654773 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:44.656656 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:44.657504 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:44.657513 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:44.658407 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:44.659627 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:44.659675 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:44.661566 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:44.661575 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:44.663884 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:44.663904 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:44.665283 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:44.665289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:44.665501 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:44.666364 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:44.666391 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:44.669225 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:44.673455 (+ 4230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:44.673466 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:44.673494 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:44.674224 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:44.674644 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:44.674649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:44.674653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:44.674856 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:44.675434 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:44.675443 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:44.675826 (+ 383us) client_negotiation.cc:769] Sending connection context
0619 06:33:44.676105 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 06:33:44.676662 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":402,"threads_started":1}
I20260619 06:33:44.705996 30908 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:44013
I20260619 06:33:44.706048 31043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:44013 every 8 connection(s)
I20260619 06:33:44.707028 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:44.691302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:45489 (local address 127.28.104.254:32843)
0619 06:33:44.691700 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:44.691709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:44.692395 (+ 686us) server_negotiation.cc:407] Connection header received
0619 06:33:44.692605 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:44.692609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:44.692673 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:44.692802 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:44.694349 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:44.695406 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:44.697314 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:44.697737 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:44.697883 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:44.700573 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:44.700585 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:44.700589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:44.700650 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:44.704121 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:44.704826 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:44.704834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:44.704839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:44.705050 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:44.705502 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:44.705505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:44.705507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:44.705766 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:44.705927 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:44.706432 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 06:33:44.706670 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260619 06:33:44.707151 31008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:44.691268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:32843 (local address 127.28.104.193:45489)
0619 06:33:44.692248 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:44.692275 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:44.692464 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:44.692847 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:44.692854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:44.693714 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:44.694186 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:44.694198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:44.695537 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:44.695546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:44.697156 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:44.697168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:44.698154 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:44.698158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:44.698285 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:44.699110 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:44.699128 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:44.700364 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:44.704251 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:44.704257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:44.704262 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:44.704688 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:44.705203 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:44.705210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:44.705219 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:44.705361 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:44.705896 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:44.705901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:44.706146 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 06:33:44.706315 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 06:33:44.706664 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":626,"threads_started":1}
I20260619 06:33:44.707417 30908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:33:44.708606 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 30908
I20260619 06:33:44.708822 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:33:44.709820 29091 external_mini_cluster.cc:1428] Setting key 378ec3a5175babb90c98d2902765f2fb
I20260619 06:33:44.710800 31044 heartbeater.cc:347] Connected to a master server at 127.28.104.254:32843
I20260619 06:33:44.711063 31044 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:44.711979 31044 heartbeater.cc:511] Master 127.28.104.254:32843 requested a full tablet report, sending...
I20260619 06:33:44.714320 30837 ts_manager.cc:194] Registered new tserver with Master: fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013)
I20260619 06:33:44.716451 30837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:58541
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:33:44.750418 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:32843
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:45.077335 31048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:45.077628 31048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:45.077731 31048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:45.085269 31048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:45.085384 31048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:45.085470 31048 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:33:45.085520 31048 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:33:45.085589 31048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:33:45.097926 31048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.104.254:32843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:45.099524 31048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:45.101151 31048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:45.116117 31053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:45.116245 31054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:45.116268 31056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:45.117059 31048 server_base.cc:1034] running on GCE node
I20260619 06:33:45.117746 31048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:45.118647 31048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:45.119892 31048 hybrid_clock.cc:648] HybridClock initialized: now 1781850825119844 us; error 57 us; skew 500 ppm
I20260619 06:33:45.120249 31048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:45 dist-test-slave-nf4n krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850825, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:45.123703 31048 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:33:45.125278 31048 webserver.cc:460] Webserver started at http://127.28.104.194:37811/ using document root <none> and password file <none>
I20260619 06:33:45.125973 31048 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:45.126109 31048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:45.126443 31048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:45.129535 31048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "d460e8fae9b045808279435753c9017a"
format_stamp: "Formatted at 2026-06-19 06:33:45 on dist-test-slave-nf4n"
server_key: "67e58a853b89a68ad273cfbc4e0aca61"
server_key_iv: "11e8d7c5f082d1fe7f911b4b1e8606c0"
server_key_version: "encryptionkey@0"
I20260619 06:33:45.130277 31048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "d460e8fae9b045808279435753c9017a"
format_stamp: "Formatted at 2026-06-19 06:33:45 on dist-test-slave-nf4n"
server_key: "67e58a853b89a68ad273cfbc4e0aca61"
server_key_iv: "11e8d7c5f082d1fe7f911b4b1e8606c0"
server_key_version: "encryptionkey@0"
I20260619 06:33:45.135356 31048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:33:45.138871 31063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:45.140166 31048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:33:45.140323 31048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "d460e8fae9b045808279435753c9017a"
format_stamp: "Formatted at 2026-06-19 06:33:45 on dist-test-slave-nf4n"
server_key: "67e58a853b89a68ad273cfbc4e0aca61"
server_key_iv: "11e8d7c5f082d1fe7f911b4b1e8606c0"
server_key_version: "encryptionkey@0"
I20260619 06:33:45.140507 31048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:45.160971 31048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:45.165181 31048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:45.165472 31048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:45.168107 31048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:45.168264 31048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:45.168403 31048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:45.168499 31048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:33:45 dist-test-slave-nf4n krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850825, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:45.197836 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.168987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:47707 (local address 127.28.104.254:32843)
0619 06:33:45.169233 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:45.169242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:45.170834 (+ 1592us) server_negotiation.cc:407] Connection header received
0619 06:33:45.174247 (+ 3413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:45.174254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:45.174375 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:45.174480 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:45.177947 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.179316 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.182915 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.183501 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.183682 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.188691 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:45.188697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:45.188700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:45.188739 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:45.192133 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:45.193394 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:45.193403 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:45.193409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:45.193599 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:45.194332 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:45.194339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:45.194346 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:45.194820 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:45.195001 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:45.197254 (+ 2253us) server_negotiation.cc:299] Negotiation successful
0619 06:33:45.197499 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260619 06:33:45.198029 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.169662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:32843 (local address 127.28.104.194:47707)
0619 06:33:45.170631 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:45.170691 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:45.173718 (+ 3027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:45.175196 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:45.175211 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:45.176342 (+ 1131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:45.177641 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.177679 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.179546 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.179580 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.182709 (+ 3129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.182728 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.183823 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.183830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.184220 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.185171 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:45.185227 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:45.188419 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:45.192281 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:45.192290 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:45.192309 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:45.193219 (+ 910us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:45.193764 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:45.193772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:45.193779 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:45.194116 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:45.194990 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:45.195000 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:45.195546 (+ 546us) client_negotiation.cc:769] Sending connection context
0619 06:33:45.195933 (+ 387us) client_negotiation.cc:240] Negotiation successful
0619 06:33:45.196401 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":434,"threads_started":1}
I20260619 06:33:45.217749 31048 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:40589
I20260619 06:33:45.217792 31182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:40589 every 8 connection(s)
I20260619 06:33:45.219326 31048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:33:45.220638 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.204274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:49643 (local address 127.28.104.254:32843)
0619 06:33:45.204517 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:45.204526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:45.205047 (+ 521us) server_negotiation.cc:407] Connection header received
0619 06:33:45.205306 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:45.205313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:45.205401 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:45.205520 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:45.206593 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.207913 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.209324 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.210111 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.210262 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.213214 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:45.213224 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:45.213232 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:45.213285 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:45.217257 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:45.218361 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:45.218366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:45.218368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:45.218525 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:45.218926 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:45.218930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:45.218932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:45.219298 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:45.219472 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:45.219970 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 06:33:45.220244 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:33:45.220945 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.204284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:32843 (local address 127.28.104.194:49643)
0619 06:33:45.204936 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:45.204953 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:45.205142 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:45.205604 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:45.205609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:45.206123 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:45.206439 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.206448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.208081 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.208087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.209117 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.209136 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.210320 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.210326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.210446 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.211175 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:45.211201 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:45.212630 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:45.217428 (+ 4798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:45.217437 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:45.217443 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:45.218232 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:45.218651 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:45.218656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:45.218659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:45.218809 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:45.219431 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:45.219435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:45.219649 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 06:33:45.219811 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 06:33:45.220282 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":352,"threads_started":1}
I20260619 06:33:45.222460 31183 heartbeater.cc:347] Connected to a master server at 127.28.104.254:32843
I20260619 06:33:45.222750 31183 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:45.223562 31183 heartbeater.cc:511] Master 127.28.104.254:32843 requested a full tablet report, sending...
I20260619 06:33:45.225307 30836 ts_manager.cc:194] Registered new tserver with Master: d460e8fae9b045808279435753c9017a (127.28.104.194:40589)
I20260619 06:33:45.226162 30836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:47707
I20260619 06:33:45.228647 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 31048
I20260619 06:33:45.228813 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:33:45.229266 29091 external_mini_cluster.cc:1428] Setting key 4dcfa0af11a38ca0f859e5966420e04b
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:33:45.270452 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:32843
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:45.608496 31187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:45.608790 31187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:45.608898 31187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:45.616463 31187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:45.616601 31187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:45.616691 31187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:33:45.616736 31187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:33:45.616858 31187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:33:45.627049 31187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.104.254:32843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:45.628579 31187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:45.629946 31187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:45.643811 31193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:45.643786 31195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:45.643764 31192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:45.644714 31187 server_base.cc:1034] running on GCE node
I20260619 06:33:45.645484 31187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:45.646420 31187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:45.647675 31187 hybrid_clock.cc:648] HybridClock initialized: now 1781850825647616 us; error 46 us; skew 500 ppm
I20260619 06:33:45.648011 31187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:45 dist-test-slave-nf4n krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850825, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:45.651165 31187 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:33:45.652654 31187 webserver.cc:460] Webserver started at http://127.28.104.195:36483/ using document root <none> and password file <none>
I20260619 06:33:45.653445 31187 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:45.653585 31187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:45.653898 31187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:45.656917 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "eaeac2cee4e14496964e89b1714e796d"
format_stamp: "Formatted at 2026-06-19 06:33:45 on dist-test-slave-nf4n"
server_key: "1de6ad647e75154e8f45bfa13e8fe5e4"
server_key_iv: "b90af4a17934fc98a9d5868eb04a7eb4"
server_key_version: "encryptionkey@0"
I20260619 06:33:45.657810 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "eaeac2cee4e14496964e89b1714e796d"
format_stamp: "Formatted at 2026-06-19 06:33:45 on dist-test-slave-nf4n"
server_key: "1de6ad647e75154e8f45bfa13e8fe5e4"
server_key_iv: "b90af4a17934fc98a9d5868eb04a7eb4"
server_key_version: "encryptionkey@0"
I20260619 06:33:45.663504 31187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:33:45.667026 31202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:45.668367 31187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:33:45.668541 31187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "eaeac2cee4e14496964e89b1714e796d"
format_stamp: "Formatted at 2026-06-19 06:33:45 on dist-test-slave-nf4n"
server_key: "1de6ad647e75154e8f45bfa13e8fe5e4"
server_key_iv: "b90af4a17934fc98a9d5868eb04a7eb4"
server_key_version: "encryptionkey@0"
I20260619 06:33:45.668757 31187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:45.692350 31187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:45.696959 31187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:45.697355 31187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:45.700356 31187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:45.700487 31187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:45.700610 31187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:45.700717 31187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:33:45 dist-test-slave-nf4n krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850825, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:45.719234 31044 heartbeater.cc:503] Master 127.28.104.254:32843 was elected leader, sending a full tablet report...
I20260619 06:33:45.725914 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.701146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:47449 (local address 127.28.104.254:32843)
0619 06:33:45.701421 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:45.701427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:45.702734 (+ 1307us) server_negotiation.cc:407] Connection header received
0619 06:33:45.704756 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:45.704763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:45.704853 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:45.704993 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:45.707570 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.708574 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.711869 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.712479 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.712581 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.718444 (+ 5863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:45.718448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:45.718452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:45.718488 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:45.721873 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:45.723035 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:45.723039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:45.723042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:45.723177 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:45.723738 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:45.723741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:45.723743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:45.724009 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:45.724141 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:45.725371 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0619 06:33:45.725560 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260619 06:33:45.728289 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.701756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:32843 (local address 127.28.104.195:47449)
0619 06:33:45.702555 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:45.702600 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:45.704456 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:45.705497 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:45.705512 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:45.706416 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:45.707378 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.707402 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.708757 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.708766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.711656 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.711679 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.712986 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.712993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.713321 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.714163 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:45.714195 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:45.718172 (+ 3977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:45.722024 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:45.722039 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:45.722063 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:45.722905 (+ 842us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:45.723291 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:45.723299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:45.723305 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:45.723622 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:45.724137 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:45.724149 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:45.724765 (+ 616us) client_negotiation.cc:769] Sending connection context
0619 06:33:45.725229 (+ 464us) client_negotiation.cc:240] Negotiation successful
0619 06:33:45.726053 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":343,"threads_started":1}
I20260619 06:33:45.751565 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.736243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:55497 (local address 127.28.104.254:32843)
0619 06:33:45.736458 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:45.736466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:45.737476 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 06:33:45.737695 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:45.737701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:45.737789 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:45.737905 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:45.738875 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.739602 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.740962 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.741565 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.741698 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.744712 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:45.744722 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:45.744731 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:45.744784 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:45.748618 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:45.749585 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:45.749593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:45.749597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:45.749734 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:45.750107 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:45.750113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:45.750117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:45.750397 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:45.750569 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:45.750953 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 06:33:45.751185 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:33:45.751644 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.736356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:32843 (local address 127.28.104.195:55497)
0619 06:33:45.737363 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:45.737381 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:45.737564 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:45.737929 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:45.737933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:45.738433 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:45.738728 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.738738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.739748 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.739754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.740803 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.740819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.741962 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.741972 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.742273 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.743063 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:45.743095 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:45.744505 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:45.748791 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:45.748795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:45.748801 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:45.749437 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:45.749849 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:45.749856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:45.749860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:45.749989 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:45.750518 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:45.750522 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:45.750704 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 06:33:45.750847 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 06:33:45.751193 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":597,"threads_started":1}
I20260619 06:33:45.762995 31187 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:34961
I20260619 06:33:45.763028 31321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:34961 every 8 connection(s)
I20260619 06:33:45.764417 31187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:33:45.766677 31322 heartbeater.cc:347] Connected to a master server at 127.28.104.254:32843
I20260619 06:33:45.766920 31322 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:45.767563 31322 heartbeater.cc:511] Master 127.28.104.254:32843 requested a full tablet report, sending...
I20260619 06:33:45.768968 30836 ts_manager.cc:194] Registered new tserver with Master: eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961)
I20260619 06:33:45.769747 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 31187
I20260619 06:33:45.769960 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:33:45.769923 30836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:47449
I20260619 06:33:45.770545 29091 external_mini_cluster.cc:1428] Setting key 37cc874e545f3f64a56f958b14a5cfce
I20260619 06:33:45.773938 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:33:45 dist-test-slave-nf4n krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:33:45 dist-test-slave-nf4n krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:45.829178 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.814572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.28.104.254:32843)
0619 06:33:45.814835 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:45.814843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:45.815109 (+ 266us) server_negotiation.cc:407] Connection header received
0619 06:33:45.815353 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:45.815360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:45.815440 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:45.815548 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:45.817125 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.817886 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.819118 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.819539 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.819682 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.823554 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:45.823561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:45.823572 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:45.823609 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:45.826293 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:45.827026 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:45.827033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:45.827037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:45.827180 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:45.827558 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:45.827564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:45.827567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:45.827886 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:45.828199 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:45.828506 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 06:33:45.828745 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 06:33:45.833736 30836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:33:45.841429 30836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:33:45.877910 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.854641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40589 (local address 127.0.0.1:58262)
0619 06:33:45.865928 (+ 11287us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:45.865988 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:45.866209 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:45.869749 (+ 3540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:45.869762 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:45.869811 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:45.870306 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.870327 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.873243 (+ 2916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.873249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.875564 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.875575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.876828 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.876834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.877064 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.877111 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 06:33:45.877283 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 06:33:45.877496 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10965,"thread_start_us":16169,"threads_started":1}
I20260619 06:33:45.878571 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.854844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.28.104.194:40589)
0619 06:33:45.868475 (+ 13631us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:45.868490 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:45.868535 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:33:45.868677 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:45.868691 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:45.869120 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:45.869394 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:45.871131 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.872949 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.875719 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.876601 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.876809 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.877000 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:45.878178 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0619 06:33:45.878356 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13381,"thread_start_us":13214,"threads_started":1}
I20260619 06:33:45.880889 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.868560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:44013 (local address 127.0.0.1:37354)
0619 06:33:45.871145 (+ 2585us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:45.871183 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:45.871847 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:45.872759 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:45.872765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:45.872815 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:45.873385 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.873397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.874940 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.874946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.877868 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.877884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.879990 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.879997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.880159 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.880201 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 06:33:45.880288 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:33:45.880476 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2431,"mutex_wait_us":145,"thread_start_us":1856,"threads_started":1}
I20260619 06:33:45.880931 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.866930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.28.104.193:44013)
0619 06:33:45.870395 (+ 3465us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:45.870407 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:45.871970 (+ 1563us) server_negotiation.cc:407] Connection header received
0619 06:33:45.872098 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:45.872106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:45.872414 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:45.872610 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:45.873579 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.874785 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.878078 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.879772 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.879968 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.880148 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:45.880386 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 06:33:45.880585 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3251,"thread_start_us":3063,"threads_started":1}
I20260619 06:33:45.882021 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.855242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.28.104.195:34961)
0619 06:33:45.871678 (+ 16436us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:45.871688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:45.871724 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:33:45.871820 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:45.871829 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:45.872119 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:45.872335 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:45.873221 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.874909 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.878634 (+ 3725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.880213 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.880447 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.880612 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:45.881474 (+ 862us) server_negotiation.cc:299] Negotiation successful
0619 06:33:45.881681 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16240,"thread_start_us":16038,"threads_started":1}
I20260619 06:33:45.882673 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.854910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:34961 (local address 127.0.0.1:34696)
0619 06:33:45.870828 (+ 15918us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:45.870854 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:45.870985 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:45.872372 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:45.872378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:45.872401 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:45.872932 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.872947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.875035 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.875043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.877877 (+ 2834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.877886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.880778 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.880785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.880996 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.881066 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 06:33:45.881165 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:33:45.882277 (+ 1112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15705,"mutex_wait_us":101,"thread_start_us":10540,"threads_started":1}
I20260619 06:33:45.882647 31113 tablet_service.cc:1467] Processing CreateTablet for tablet 66fac7f719764095a0829f329c69e4a0 (DEFAULT_TABLE table=test-table [id=fe38a1be78c04861bb5771feb0501f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:45.883920 31113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66fac7f719764095a0829f329c69e4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:45.884809 30974 tablet_service.cc:1467] Processing CreateTablet for tablet 66fac7f719764095a0829f329c69e4a0 (DEFAULT_TABLE table=test-table [id=fe38a1be78c04861bb5771feb0501f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:45.886375 30974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66fac7f719764095a0829f329c69e4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:45.888763 31252 tablet_service.cc:1467] Processing CreateTablet for tablet 66fac7f719764095a0829f329c69e4a0 (DEFAULT_TABLE table=test-table [id=fe38a1be78c04861bb5771feb0501f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:45.890319 31252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66fac7f719764095a0829f329c69e4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:45.895996 31341 tablet_bootstrap.cc:492] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: Bootstrap starting.
I20260619 06:33:45.899144 31341 tablet_bootstrap.cc:654] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:45.900413 31341 log.cc:824] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:45.902369 31343 tablet_bootstrap.cc:492] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: Bootstrap starting.
I20260619 06:33:45.903829 31341 tablet_bootstrap.cc:492] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: No bootstrap required, opened a new log
I20260619 06:33:45.904127 31341 ts_tablet_manager.cc:1397] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260619 06:33:45.906008 31343 tablet_bootstrap.cc:654] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:45.907227 31343 log.cc:824] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:45.907264 31344 tablet_bootstrap.cc:492] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: Bootstrap starting.
I20260619 06:33:45.910698 31343 tablet_bootstrap.cc:492] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: No bootstrap required, opened a new log
I20260619 06:33:45.910969 31343 ts_tablet_manager.cc:1397] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 06:33:45.910989 31344 tablet_bootstrap.cc:654] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:45.912397 31344 log.cc:824] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:45.914310 31341 raft_consensus.cc:348] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } }
I20260619 06:33:45.914764 31341 raft_consensus.cc:374] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:45.914866 31341 raft_consensus.cc:729] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d460e8fae9b045808279435753c9017a, State: Initialized, Role: FOLLOWER
I20260619 06:33:45.915390 31344 tablet_bootstrap.cc:492] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: No bootstrap required, opened a new log
I20260619 06:33:45.915690 31344 ts_tablet_manager.cc:1397] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260619 06:33:45.915520 31341 consensus_queue.cc:260] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } }
I20260619 06:33:45.916790 31183 heartbeater.cc:503] Master 127.28.104.254:32843 was elected leader, sending a full tablet report...
I20260619 06:33:45.917472 31341 ts_tablet_manager.cc:1428] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: Time spent starting tablet: real 0.013s user 0.012s sys 0.003s
I20260619 06:33:45.921566 31343 raft_consensus.cc:348] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } }
I20260619 06:33:45.922014 31343 raft_consensus.cc:374] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:45.922122 31343 raft_consensus.cc:729] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd77a969ff064b849ebd584641a116f1, State: Initialized, Role: FOLLOWER
I20260619 06:33:45.922755 31343 consensus_queue.cc:260] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } }
I20260619 06:33:45.924695 31343 ts_tablet_manager.cc:1428] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260619 06:33:45.926874 31344 raft_consensus.cc:348] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } }
I20260619 06:33:45.927321 31344 raft_consensus.cc:374] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:45.927444 31344 raft_consensus.cc:729] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaeac2cee4e14496964e89b1714e796d, State: Initialized, Role: FOLLOWER
I20260619 06:33:45.927995 31344 consensus_queue.cc:260] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } }
I20260619 06:33:45.929180 31322 heartbeater.cc:503] Master 127.28.104.254:32843 was elected leader, sending a full tablet report...
I20260619 06:33:45.929661 31344 ts_tablet_manager.cc:1428] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
W20260619 06:33:45.964491 31045 tablet.cc:2369] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:33:45.967481 31349 raft_consensus.cc:482] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:33:45.967790 31349 raft_consensus.cc:504] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } }
I20260619 06:33:45.969520 31349 leader_election.cc:288] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d460e8fae9b045808279435753c9017a (127.28.104.194:40589), fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013)
W20260619 06:33:45.972919 31184 tablet.cc:2369] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:33:45.975831 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.970103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:44013 (local address 127.28.104.195:53747)
0619 06:33:45.970284 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:45.970305 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:45.970440 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:45.970790 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:45.970795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:45.970826 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:45.971047 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.971100 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.972317 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.972321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.974176 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.974191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.975312 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.975316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.975442 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.975465 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:33:45.975538 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 06:33:45.975627 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 06:33:45.975960 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.970071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:53747 (local address 127.28.104.193:44013)
0619 06:33:45.970284 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:45.970290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:45.970398 (+ 108us) server_negotiation.cc:407] Connection header received
0619 06:33:45.970555 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:45.970577 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:45.970642 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:45.970767 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:45.971251 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.972183 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.974336 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.975173 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.975329 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.975408 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:45.975591 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 06:33:45.975723 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:33:45.977097 30998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fac7f719764095a0829f329c69e4a0" candidate_uuid: "eaeac2cee4e14496964e89b1714e796d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd77a969ff064b849ebd584641a116f1" is_pre_election: true
I20260619 06:33:45.977334 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.970119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:37037 (local address 127.28.104.194:40589)
0619 06:33:45.970302 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:45.970310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:45.971396 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 06:33:45.971497 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:45.971502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:45.971582 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:45.971674 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:45.972245 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.973314 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.975170 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.976379 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.976543 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.976609 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:45.977001 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 06:33:45.977169 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:33:45.977574 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:45.970120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40589 (local address 127.28.104.195:37037)
0619 06:33:45.971287 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:45.971301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:45.971400 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:45.971727 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:45.971730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:45.971752 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:45.972082 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.972091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.973437 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.973443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.975018 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:45.975029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:45.976627 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:45.976635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:45.976832 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:45.976864 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:33:45.976971 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 06:33:45.977189 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":397,"thread_start_us":410,"threads_started":1}
I20260619 06:33:45.977718 30998 raft_consensus.cc:2454] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaeac2cee4e14496964e89b1714e796d in term 0.
I20260619 06:33:45.978575 31204 leader_election.cc:302] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaeac2cee4e14496964e89b1714e796d, fd77a969ff064b849ebd584641a116f1; no voters:
I20260619 06:33:45.978696 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fac7f719764095a0829f329c69e4a0" candidate_uuid: "eaeac2cee4e14496964e89b1714e796d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d460e8fae9b045808279435753c9017a" is_pre_election: true
I20260619 06:33:45.979043 31349 raft_consensus.cc:2781] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:33:45.979243 31349 raft_consensus.cc:482] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:33:45.979256 31133 raft_consensus.cc:2454] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaeac2cee4e14496964e89b1714e796d in term 0.
I20260619 06:33:45.979358 31349 raft_consensus.cc:3037] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:45.981859 31349 raft_consensus.cc:504] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } }
I20260619 06:33:45.982445 31349 leader_election.cc:288] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [CANDIDATE]: Term 1 election: Requested vote from peers d460e8fae9b045808279435753c9017a (127.28.104.194:40589), fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013)
I20260619 06:33:45.982932 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fac7f719764095a0829f329c69e4a0" candidate_uuid: "eaeac2cee4e14496964e89b1714e796d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d460e8fae9b045808279435753c9017a"
I20260619 06:33:45.982971 30998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fac7f719764095a0829f329c69e4a0" candidate_uuid: "eaeac2cee4e14496964e89b1714e796d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd77a969ff064b849ebd584641a116f1"
I20260619 06:33:45.983129 31133 raft_consensus.cc:3037] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:45.983214 30998 raft_consensus.cc:3037] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:45.985431 31133 raft_consensus.cc:2454] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaeac2cee4e14496964e89b1714e796d in term 1.
I20260619 06:33:45.985473 30998 raft_consensus.cc:2454] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaeac2cee4e14496964e89b1714e796d in term 1.
I20260619 06:33:45.985862 31206 leader_election.cc:302] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d460e8fae9b045808279435753c9017a, eaeac2cee4e14496964e89b1714e796d; no voters:
I20260619 06:33:45.986240 31349 raft_consensus.cc:2781] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:45.986693 31349 raft_consensus.cc:686] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 1 LEADER]: Becoming Leader. State: Replica: eaeac2cee4e14496964e89b1714e796d, State: Running, Role: LEADER
I20260619 06:33:45.987161 31349 consensus_queue.cc:237] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } }
I20260619 06:33:45.992319 30836 catalog_manager.cc:5495] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d reported cstate change: term changed from 0 to 1, leader changed from <none> to eaeac2cee4e14496964e89b1714e796d (127.28.104.195). New cstate: current_term: 1 leader_uuid: "eaeac2cee4e14496964e89b1714e796d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } health_report { overall_health: UNKNOWN } } }
W20260619 06:33:46.016551 31323 tablet.cc:2369] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:33:46 dist-test-slave-nf4n krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850824, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:46.049762 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.035721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:32843 (local address 127.0.0.1:38716)
0619 06:33:46.036325 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:46.036363 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:46.036540 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:46.036865 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:46.036869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:46.037689 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:46.037964 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.037972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.038943 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.038951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.039911 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.039922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.040643 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.040649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.040819 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.041496 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:46.041512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:46.043807 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:46.047200 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:46.047211 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:46.047219 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:46.047617 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:46.047970 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:46.047977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:46.047980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:46.048109 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:46.048616 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:46.048624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:46.048787 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 06:33:46.048948 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 06:33:46.049262 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":316,"threads_started":1}
I20260619 06:33:46.049759 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.035582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.28.104.254:32843)
0619 06:33:46.035769 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:46.035773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:46.036460 (+ 687us) server_negotiation.cc:407] Connection header received
0619 06:33:46.036664 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:46.036669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:46.036738 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:46.036840 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:46.038100 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.038820 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.040044 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.040462 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.040600 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.044026 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:46.044031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:46.044034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:46.044156 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:46.047054 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:46.047735 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:46.047742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:46.047745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:46.047870 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:46.048216 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:46.048224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:46.048227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:46.048488 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:46.048629 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:46.049056 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 06:33:46.049397 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":56}
I20260619 06:33:46.054807 30836 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:38716:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:33:46.055799 30836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:33:46.062598 30974 tablet_service.cc:1467] Processing CreateTablet for tablet 85182a91d6b04c579f383383eb2da3dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b37a77bed9184f3596b2be8f6024186e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:33:46.062700 31113 tablet_service.cc:1467] Processing CreateTablet for tablet 85182a91d6b04c579f383383eb2da3dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b37a77bed9184f3596b2be8f6024186e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:33:46.062949 31252 tablet_service.cc:1467] Processing CreateTablet for tablet 85182a91d6b04c579f383383eb2da3dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b37a77bed9184f3596b2be8f6024186e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:33:46.063606 31113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85182a91d6b04c579f383383eb2da3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:46.063606 30974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85182a91d6b04c579f383383eb2da3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:46.063671 31252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85182a91d6b04c579f383383eb2da3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:46.069689 31343 tablet_bootstrap.cc:492] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: Bootstrap starting.
I20260619 06:33:46.070853 31341 tablet_bootstrap.cc:492] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: Bootstrap starting.
I20260619 06:33:46.071224 31343 tablet_bootstrap.cc:654] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:46.071605 31344 tablet_bootstrap.cc:492] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: Bootstrap starting.
I20260619 06:33:46.072705 31341 tablet_bootstrap.cc:654] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:46.073571 31344 tablet_bootstrap.cc:654] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:46.073961 31343 tablet_bootstrap.cc:492] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: No bootstrap required, opened a new log
I20260619 06:33:46.074131 31343 ts_tablet_manager.cc:1397] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 06:33:46.074970 31343 raft_consensus.cc:348] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.075241 31343 raft_consensus.cc:374] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:46.075181 31341 tablet_bootstrap.cc:492] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: No bootstrap required, opened a new log
I20260619 06:33:46.075327 31343 raft_consensus.cc:729] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd77a969ff064b849ebd584641a116f1, State: Initialized, Role: FOLLOWER
I20260619 06:33:46.075527 31341 ts_tablet_manager.cc:1397] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260619 06:33:46.075651 31343 consensus_queue.cc:260] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.075815 31344 tablet_bootstrap.cc:492] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: No bootstrap required, opened a new log
I20260619 06:33:46.075943 31344 ts_tablet_manager.cc:1397] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 06:33:46.076152 31348 tablet_replica.cc:439] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } }
I20260619 06:33:46.076503 31348 tablet_replica.cc:442] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:46.076565 31341 raft_consensus.cc:348] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.076790 31341 raft_consensus.cc:374] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:46.076879 31341 raft_consensus.cc:729] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d460e8fae9b045808279435753c9017a, State: Initialized, Role: FOLLOWER
I20260619 06:33:46.076759 31344 raft_consensus.cc:348] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.077047 31344 raft_consensus.cc:374] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:46.077142 31344 raft_consensus.cc:729] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaeac2cee4e14496964e89b1714e796d, State: Initialized, Role: FOLLOWER
I20260619 06:33:46.077198 31341 consensus_queue.cc:260] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.077366 31344 consensus_queue.cc:260] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.077776 31349 tablet_replica.cc:439] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } }
I20260619 06:33:46.077714 31347 tablet_replica.cc:439] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } }
I20260619 06:33:46.078068 31349 tablet_replica.cc:442] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:46.078071 31347 tablet_replica.cc:442] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:46.078428 31341 ts_tablet_manager.cc:1428] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260619 06:33:46.078538 31344 ts_tablet_manager.cc:1428] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 06:33:46.079097 31343 ts_tablet_manager.cc:1428] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 06:33:46.258898 31349 raft_consensus.cc:482] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:33:46.259131 31349 raft_consensus.cc:504] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.259757 31349 leader_election.cc:288] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013), d460e8fae9b045808279435753c9017a (127.28.104.194:40589)
I20260619 06:33:46.260393 30998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85182a91d6b04c579f383383eb2da3dd" candidate_uuid: "eaeac2cee4e14496964e89b1714e796d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd77a969ff064b849ebd584641a116f1" is_pre_election: true
I20260619 06:33:46.260465 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85182a91d6b04c579f383383eb2da3dd" candidate_uuid: "eaeac2cee4e14496964e89b1714e796d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d460e8fae9b045808279435753c9017a" is_pre_election: true
I20260619 06:33:46.260640 30998 raft_consensus.cc:2454] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaeac2cee4e14496964e89b1714e796d in term 0.
I20260619 06:33:46.260828 31133 raft_consensus.cc:2454] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaeac2cee4e14496964e89b1714e796d in term 0.
I20260619 06:33:46.261125 31204 leader_election.cc:302] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaeac2cee4e14496964e89b1714e796d, fd77a969ff064b849ebd584641a116f1; no voters:
I20260619 06:33:46.261427 31349 raft_consensus.cc:2781] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:33:46.261518 31349 raft_consensus.cc:482] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:33:46.261679 31349 raft_consensus.cc:3037] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:46.263806 31349 raft_consensus.cc:504] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.264550 31349 leader_election.cc:288] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [CANDIDATE]: Term 1 election: Requested vote from peers fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013), d460e8fae9b045808279435753c9017a (127.28.104.194:40589)
I20260619 06:33:46.264948 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85182a91d6b04c579f383383eb2da3dd" candidate_uuid: "eaeac2cee4e14496964e89b1714e796d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d460e8fae9b045808279435753c9017a"
I20260619 06:33:46.264952 30998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85182a91d6b04c579f383383eb2da3dd" candidate_uuid: "eaeac2cee4e14496964e89b1714e796d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd77a969ff064b849ebd584641a116f1"
I20260619 06:33:46.265213 31133 raft_consensus.cc:3037] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:46.265242 30998 raft_consensus.cc:3037] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:46.267151 31133 raft_consensus.cc:2454] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaeac2cee4e14496964e89b1714e796d in term 1.
I20260619 06:33:46.267380 30998 raft_consensus.cc:2454] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaeac2cee4e14496964e89b1714e796d in term 1.
I20260619 06:33:46.267577 31206 leader_election.cc:302] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d460e8fae9b045808279435753c9017a, eaeac2cee4e14496964e89b1714e796d; no voters:
I20260619 06:33:46.267853 31349 raft_consensus.cc:2781] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:46.268016 31349 raft_consensus.cc:686] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [term 1 LEADER]: Becoming Leader. State: Replica: eaeac2cee4e14496964e89b1714e796d, State: Running, Role: LEADER
I20260619 06:33:46.268288 31349 consensus_queue.cc:237] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.269738 31351 tablet_replica.cc:439] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: TxnStatusTablet state changed. Reason: New leader eaeac2cee4e14496964e89b1714e796d. Latest consensus state: current_term: 1 leader_uuid: "eaeac2cee4e14496964e89b1714e796d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } }
I20260619 06:33:46.269982 31351 tablet_replica.cc:442] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:33:46.270570 31363 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:33:46.270835 31363 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:33:46.271335 30836 catalog_manager.cc:5495] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d reported cstate change: term changed from 0 to 1, leader changed from <none> to eaeac2cee4e14496964e89b1714e796d (127.28.104.195). New cstate: current_term: 1 leader_uuid: "eaeac2cee4e14496964e89b1714e796d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } health_report { overall_health: HEALTHY } } }
I20260619 06:33:46.314410 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.308290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.28.104.195:34961)
0619 06:33:46.308485 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:46.308492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:46.308685 (+ 193us) server_negotiation.cc:407] Connection header received
0619 06:33:46.308983 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:46.308987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:46.309140 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:46.309286 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:46.309702 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.310433 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.311876 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.312504 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.312646 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.313175 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:46.313792 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:46.313941 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:46.314024 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 06:33:46.314103 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:33:46.314502 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.308345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:34961 (local address 127.0.0.1:34704)
0619 06:33:46.308581 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:46.308597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:46.308762 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:46.309307 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:46.309312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:46.309323 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:33:46.309512 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.309540 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.310566 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.310574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.311740 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.311751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.312679 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.312685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.312869 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.312932 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:33:46.313896 (+ 964us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:33:46.313918 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:33:46.314003 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:33:46.314153 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 06:33:46.322474 31133 raft_consensus.cc:1261] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Refusing update from remote peer eaeac2cee4e14496964e89b1714e796d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:46.322474 30998 raft_consensus.cc:1261] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Refusing update from remote peer eaeac2cee4e14496964e89b1714e796d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:46.323580 31351 consensus_queue.cc:1035] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:46.323972 31349 consensus_queue.cc:1035] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:46.336016 31347 tablet_replica.cc:439] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: TxnStatusTablet state changed. Reason: New leader eaeac2cee4e14496964e89b1714e796d. Latest consensus state: current_term: 1 leader_uuid: "eaeac2cee4e14496964e89b1714e796d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } }
I20260619 06:33:46.336385 31347 tablet_replica.cc:442] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:46.339686 31366 mvcc.cc:205] Tried to move back new op lower bound from 7298460984606281728 to 7298460984397099008. Current Snapshot: MvccSnapshot[applied={T|T < 7298460984606281728}]
I20260619 06:33:46.340134 31351 tablet_replica.cc:439] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eaeac2cee4e14496964e89b1714e796d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } }
I20260619 06:33:46.340382 31351 tablet_replica.cc:442] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:33:46.340672 31367 mvcc.cc:205] Tried to move back new op lower bound from 7298460984606281728 to 7298460984397099008. Current Snapshot: MvccSnapshot[applied={T|T < 7298460984606281728}]
I20260619 06:33:46.341629 31347 tablet_replica.cc:439] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eaeac2cee4e14496964e89b1714e796d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } }
I20260619 06:33:46.341878 31347 tablet_replica.cc:442] T 85182a91d6b04c579f383383eb2da3dd P d460e8fae9b045808279435753c9017a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:46.352875 31349 tablet_replica.cc:439] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eaeac2cee4e14496964e89b1714e796d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } }
I20260619 06:33:46.353286 31349 tablet_replica.cc:442] T 85182a91d6b04c579f383383eb2da3dd P eaeac2cee4e14496964e89b1714e796d: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:33:46.357193 31373 tablet_replica.cc:439] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eaeac2cee4e14496964e89b1714e796d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } }
I20260619 06:33:46.357359 31348 tablet_replica.cc:439] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: TxnStatusTablet state changed. Reason: New leader eaeac2cee4e14496964e89b1714e796d. Latest consensus state: current_term: 1 leader_uuid: "eaeac2cee4e14496964e89b1714e796d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } } }
I20260619 06:33:46.357649 31373 tablet_replica.cc:442] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:46.357699 31348 tablet_replica.cc:442] T 85182a91d6b04c579f383383eb2da3dd P fd77a969ff064b849ebd584641a116f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:33:46.360632 31368 mvcc.cc:205] Tried to move back new op lower bound from 7298460984606281728 to 7298460984397099008. Current Snapshot: MvccSnapshot[applied={T|T < 7298460984606281728 or (T in {7298460984606281728})}]
I20260619 06:33:46.368462 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.362742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.28.104.195:34961)
0619 06:33:46.362941 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:46.362949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:46.363472 (+ 523us) server_negotiation.cc:407] Connection header received
0619 06:33:46.363700 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:46.363706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:46.363802 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:46.363923 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:46.364388 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.365272 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.366664 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.367123 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.367275 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.367701 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:46.367816 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:46.367921 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:46.368079 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 06:33:46.368161 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:33:46.387466 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.380616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:34961 (local address 127.28.104.195:58259)
0619 06:33:46.381418 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:46.381434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:46.381596 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:46.381984 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:46.381987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:46.381997 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:33:46.382312 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.382319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.383910 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.383914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.385163 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.385176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.386277 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.386281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.386393 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.386449 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:33:46.386964 (+ 515us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:33:46.386972 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:33:46.387031 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 06:33:46.387124 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":498,"threads_started":1}
I20260619 06:33:46.387470 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.380514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:58259 (local address 127.28.104.195:34961)
0619 06:33:46.380760 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:46.380767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:46.381542 (+ 775us) server_negotiation.cc:407] Connection header received
0619 06:33:46.381752 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:46.381758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:46.381860 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:46.381957 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:46.382473 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.383769 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.385304 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.386095 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.386241 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.386697 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:46.386860 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:46.386963 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:46.387083 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 06:33:46.387163 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 06:33:46.398164 30998 raft_consensus.cc:1261] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Refusing update from remote peer eaeac2cee4e14496964e89b1714e796d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:46.398218 31132 raft_consensus.cc:1261] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Refusing update from remote peer eaeac2cee4e14496964e89b1714e796d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:46.398765 31349 consensus_queue.cc:1035] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:46.399123 31380 consensus_queue.cc:1035] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:46.470985 30837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:46.471292 30837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:46.475133 30837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5: Sending DeleteTablet for 3 replicas of tablet 66fac7f719764095a0829f329c69e4a0
I20260619 06:33:46.476284 31113 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fac7f719764095a0829f329c69e4a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:46 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:58262
I20260619 06:33:46.476639 31252 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fac7f719764095a0829f329c69e4a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:46 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:34696
I20260619 06:33:46.477104 30974 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fac7f719764095a0829f329c69e4a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:46 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:37354
I20260619 06:33:46.478134 31386 tablet_replica.cc:330] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: stopping tablet replica
I20260619 06:33:46.478415 30837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:33:46.478628 31386 raft_consensus.cc:2229] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 1 LEADER]: Raft consensus shutting down.
W20260619 06:33:46.479084 30837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:33:46.479542 31386 raft_consensus.cc:2258] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:46.481680 31386 ts_tablet_manager.cc:1905] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:46.485505 31387 tablet_replica.cc:330] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: stopping tablet replica
I20260619 06:33:46.486034 31387 raft_consensus.cc:2229] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:46.486433 31387 raft_consensus.cc:2258] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:46.490257 31386 ts_tablet_manager.cc:1918] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:33:46.490469 31386 log.cc:1196] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/66fac7f719764095a0829f329c69e4a0
I20260619 06:33:46.491444 31112 tablet_service.cc:1467] Processing CreateTablet for tablet 7492bd636eee40559d8753a55a6ccc01 (DEFAULT_TABLE table=test-table [id=a7e8ece2972749699444a891282566d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:46.492026 31112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7492bd636eee40559d8753a55a6ccc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:46.492405 31386 ts_tablet_manager.cc:1939] T 66fac7f719764095a0829f329c69e4a0 P eaeac2cee4e14496964e89b1714e796d: Deleting consensus metadata
I20260619 06:33:46.492952 31387 ts_tablet_manager.cc:1905] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:46.495412 30822 catalog_manager.cc:4841] TS eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961): tablet 66fac7f719764095a0829f329c69e4a0 (table test-table [id=fe38a1be78c04861bb5771feb0501f07]) successfully deleted
I20260619 06:33:46.497462 30974 tablet_service.cc:1467] Processing CreateTablet for tablet 7492bd636eee40559d8753a55a6ccc01 (DEFAULT_TABLE table=test-table [id=a7e8ece2972749699444a891282566d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:46.497982 30974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7492bd636eee40559d8753a55a6ccc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:46.500545 31252 tablet_service.cc:1467] Processing CreateTablet for tablet 7492bd636eee40559d8753a55a6ccc01 (DEFAULT_TABLE table=test-table [id=a7e8ece2972749699444a891282566d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:46.503665 31341 tablet_bootstrap.cc:492] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a: Bootstrap starting.
I20260619 06:33:46.504338 31385 tablet_replica.cc:330] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: stopping tablet replica
I20260619 06:33:46.504884 31385 raft_consensus.cc:2229] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:46.505412 31385 raft_consensus.cc:2258] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:46.505767 31341 tablet_bootstrap.cc:654] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:46.507414 31387 ts_tablet_manager.cc:1918] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:33:46.507616 31387 log.cc:1196] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/66fac7f719764095a0829f329c69e4a0
I20260619 06:33:46.507776 31385 ts_tablet_manager.cc:1905] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:46.508189 31387 ts_tablet_manager.cc:1939] T 66fac7f719764095a0829f329c69e4a0 P fd77a969ff064b849ebd584641a116f1: Deleting consensus metadata
I20260619 06:33:46.510082 30822 catalog_manager.cc:4841] TS fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013): tablet 66fac7f719764095a0829f329c69e4a0 (table test-table [id=fe38a1be78c04861bb5771feb0501f07]) successfully deleted
I20260619 06:33:46.510335 31252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7492bd636eee40559d8753a55a6ccc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:46.516134 31385 ts_tablet_manager.cc:1918] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:33:46.516410 31385 log.cc:1196] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/66fac7f719764095a0829f329c69e4a0
I20260619 06:33:46.517082 31385 ts_tablet_manager.cc:1939] T 66fac7f719764095a0829f329c69e4a0 P d460e8fae9b045808279435753c9017a: Deleting consensus metadata
I20260619 06:33:46.518556 31344 tablet_bootstrap.cc:492] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d: Bootstrap starting.
I20260619 06:33:46.519047 31343 tablet_bootstrap.cc:492] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1: Bootstrap starting.
I20260619 06:33:46.520359 31344 tablet_bootstrap.cc:654] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:46.520751 31343 tablet_bootstrap.cc:654] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:46.523944 31341 tablet_bootstrap.cc:492] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a: No bootstrap required, opened a new log
I20260619 06:33:46.524152 31341 ts_tablet_manager.cc:1397] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.006s
I20260619 06:33:46.526494 31341 raft_consensus.cc:348] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.526599 30824 catalog_manager.cc:4841] TS d460e8fae9b045808279435753c9017a (127.28.104.194:40589): tablet 66fac7f719764095a0829f329c69e4a0 (table test-table [id=fe38a1be78c04861bb5771feb0501f07]) successfully deleted
I20260619 06:33:46.526777 31341 raft_consensus.cc:374] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:46.526856 31341 raft_consensus.cc:729] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d460e8fae9b045808279435753c9017a, State: Initialized, Role: FOLLOWER
I20260619 06:33:46.527189 31341 consensus_queue.cc:260] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.530540 31341 ts_tablet_manager.cc:1428] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260619 06:33:46.533380 31343 tablet_bootstrap.cc:492] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1: No bootstrap required, opened a new log
I20260619 06:33:46.533577 31343 ts_tablet_manager.cc:1397] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20260619 06:33:46.533845 31344 tablet_bootstrap.cc:492] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d: No bootstrap required, opened a new log
I20260619 06:33:46.534404 31343 raft_consensus.cc:348] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.534624 31343 raft_consensus.cc:374] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:46.534734 31343 raft_consensus.cc:729] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd77a969ff064b849ebd584641a116f1, State: Initialized, Role: FOLLOWER
I20260619 06:33:46.534971 31344 ts_tablet_manager.cc:1397] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.009s
I20260619 06:33:46.534998 31343 consensus_queue.cc:260] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.535596 31343 ts_tablet_manager.cc:1428] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 06:33:46.535910 31344 raft_consensus.cc:348] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.537292 31344 raft_consensus.cc:374] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:46.537413 31344 raft_consensus.cc:729] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaeac2cee4e14496964e89b1714e796d, State: Initialized, Role: FOLLOWER
I20260619 06:33:46.537680 31344 consensus_queue.cc:260] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.538364 31344 ts_tablet_manager.cc:1428] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 06:33:46.599365 31348 raft_consensus.cc:482] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:33:46.599581 31348 raft_consensus.cc:504] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.601251 31348 leader_election.cc:288] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d460e8fae9b045808279435753c9017a (127.28.104.194:40589), eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961)
I20260619 06:33:46.609684 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.601878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:54525 (local address 127.28.104.195:34961)
0619 06:33:46.602156 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:46.602164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:46.603036 (+ 872us) server_negotiation.cc:407] Connection header received
0619 06:33:46.603322 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:46.603329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:46.603435 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:46.603547 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:46.604500 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.605490 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.607364 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.608592 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.608766 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.608843 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:46.609221 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 06:33:46.609360 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 06:33:46.609819 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.601654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:34961 (local address 127.28.104.193:54525)
0619 06:33:46.602921 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:46.602946 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:46.603133 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:46.603736 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:46.603741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:46.603773 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:46.604291 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.604307 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.605635 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.605641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.607219 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.607231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.608777 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.608784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.608981 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.609048 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 06:33:46.609169 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 06:33:46.609329 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":64,"thread_start_us":657,"threads_started":1}
I20260619 06:33:46.610641 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.601857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:48167 (local address 127.28.104.194:40589)
0619 06:33:46.603223 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:46.603229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:46.603251 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:33:46.603322 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:46.603325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:46.603380 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:46.603487 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:46.604001 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.604973 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.607832 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.609062 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.609242 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.609310 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:46.610198 (+ 888us) server_negotiation.cc:299] Negotiation successful
0619 06:33:46.610334 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1227,"thread_start_us":1124,"threads_started":1}
I20260619 06:33:46.610679 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.601466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40589 (local address 127.28.104.193:48167)
0619 06:33:46.602583 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:46.602608 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:46.602787 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:46.603510 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:46.603514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:46.603543 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:46.603825 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.603835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.605134 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.605141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.607665 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.607681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.609478 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.609485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.609663 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.609722 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 06:33:46.610163 (+ 441us) client_negotiation.cc:240] Negotiation successful
0619 06:33:46.610303 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":1,"thread_start_us":1149,"threads_started":1}
I20260619 06:33:46.610844 31277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7492bd636eee40559d8753a55a6ccc01" candidate_uuid: "fd77a969ff064b849ebd584641a116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaeac2cee4e14496964e89b1714e796d" is_pre_election: true
I20260619 06:33:46.611387 31277 raft_consensus.cc:2454] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd77a969ff064b849ebd584641a116f1 in term 0.
I20260619 06:33:46.611615 31132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7492bd636eee40559d8753a55a6ccc01" candidate_uuid: "fd77a969ff064b849ebd584641a116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d460e8fae9b045808279435753c9017a" is_pre_election: true
I20260619 06:33:46.611905 31132 raft_consensus.cc:2454] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd77a969ff064b849ebd584641a116f1 in term 0.
I20260619 06:33:46.612100 30925 leader_election.cc:302] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaeac2cee4e14496964e89b1714e796d, fd77a969ff064b849ebd584641a116f1; no voters:
I20260619 06:33:46.612593 31348 raft_consensus.cc:2781] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:33:46.612735 31348 raft_consensus.cc:482] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:33:46.612846 31348 raft_consensus.cc:3037] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:46.615272 31348 raft_consensus.cc:504] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.615892 31348 leader_election.cc:288] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [CANDIDATE]: Term 1 election: Requested vote from peers d460e8fae9b045808279435753c9017a (127.28.104.194:40589), eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961)
I20260619 06:33:46.616339 31132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7492bd636eee40559d8753a55a6ccc01" candidate_uuid: "fd77a969ff064b849ebd584641a116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d460e8fae9b045808279435753c9017a"
I20260619 06:33:46.616460 31277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7492bd636eee40559d8753a55a6ccc01" candidate_uuid: "fd77a969ff064b849ebd584641a116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaeac2cee4e14496964e89b1714e796d"
I20260619 06:33:46.616588 31132 raft_consensus.cc:3037] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:46.616670 31277 raft_consensus.cc:3037] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:46.619910 31132 raft_consensus.cc:2454] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd77a969ff064b849ebd584641a116f1 in term 1.
I20260619 06:33:46.619932 31277 raft_consensus.cc:2454] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd77a969ff064b849ebd584641a116f1 in term 1.
I20260619 06:33:46.620374 30927 leader_election.cc:302] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d460e8fae9b045808279435753c9017a, fd77a969ff064b849ebd584641a116f1; no voters:
I20260619 06:33:46.620754 31348 raft_consensus.cc:2781] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:46.620935 31348 raft_consensus.cc:686] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 1 LEADER]: Becoming Leader. State: Replica: fd77a969ff064b849ebd584641a116f1, State: Running, Role: LEADER
I20260619 06:33:46.621364 31348 consensus_queue.cc:237] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:46.625157 30837 catalog_manager.cc:5495] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd77a969ff064b849ebd584641a116f1 (127.28.104.193). New cstate: current_term: 1 leader_uuid: "fd77a969ff064b849ebd584641a116f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } health_report { overall_health: UNKNOWN } } }
I20260619 06:33:46.669265 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.663308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.28.104.193:44013)
0619 06:33:46.663912 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:46.663916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:46.663932 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:33:46.664010 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:46.664014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:46.664083 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:46.664258 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:46.664666 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.665642 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.666943 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.667364 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.667546 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.667978 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:46.668556 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:46.668736 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:46.668784 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 06:33:46.668861 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":348,"threads_started":1}
I20260619 06:33:46.687422 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.680566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:37105 (local address 127.28.104.195:34961)
0619 06:33:46.680810 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:46.680820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:46.681441 (+ 621us) server_negotiation.cc:407] Connection header received
0619 06:33:46.681649 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:46.681653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:46.681730 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:46.681807 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:46.682434 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.683365 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.685627 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.686058 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.686146 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.686771 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:46.686874 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:46.686941 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:46.687119 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 06:33:46.687187 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:33:46.687871 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.680517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:34961 (local address 127.28.104.193:37105)
0619 06:33:46.681330 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:46.681352 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:46.681519 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:46.681865 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:46.681873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:46.681888 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:33:46.682265 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.682280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.683508 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.683514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.685489 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.685506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.686270 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.686280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.686490 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.686574 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:33:46.686988 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:33:46.687001 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 06:33:46.687103 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 06:33:46.687289 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":408,"threads_started":1}
I20260619 06:33:46.697181 31132 raft_consensus.cc:1261] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Refusing update from remote peer fd77a969ff064b849ebd584641a116f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:46.697365 31277 raft_consensus.cc:1261] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Refusing update from remote peer fd77a969ff064b849ebd584641a116f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:46.697873 31373 consensus_queue.cc:1035] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:46.698215 31348 consensus_queue.cc:1035] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:46.737316 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.731394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:39013 (local address 127.28.104.193:44013)
0619 06:33:46.731576 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:46.731583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:46.731601 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:33:46.731665 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:46.731670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:46.731758 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:46.731855 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:46.732460 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.733402 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.735316 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.735957 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.736103 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.736585 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:46.736681 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:46.736783 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:46.736940 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 06:33:46.737014 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:33:46.737368 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:46.730661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:44013 (local address 127.28.104.195:39013)
0619 06:33:46.731008 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:46.731027 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:46.731173 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:46.732047 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:46.732052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:46.732064 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:33:46.732284 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.732298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.733539 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.733544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.735148 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:46.735160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:46.736222 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:46.736226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:46.736370 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:46.736406 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:33:46.736788 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:33:46.736802 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 06:33:46.736910 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 06:33:46.737068 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
W20260619 06:33:46.738698 31368 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:33:48.349328 30837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:48.349588 30837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:48.353463 30837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5: Sending DeleteTablet for 3 replicas of tablet 7492bd636eee40559d8753a55a6ccc01
I20260619 06:33:48.354307 30974 tablet_service.cc:1514] Processing DeleteTablet for tablet 7492bd636eee40559d8753a55a6ccc01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:48 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:37354
I20260619 06:33:48.354348 31112 tablet_service.cc:1514] Processing DeleteTablet for tablet 7492bd636eee40559d8753a55a6ccc01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:48 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:58262
I20260619 06:33:48.354445 31252 tablet_service.cc:1514] Processing DeleteTablet for tablet 7492bd636eee40559d8753a55a6ccc01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:48 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:34696
I20260619 06:33:48.366871 31426 tablet_replica.cc:330] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1: stopping tablet replica
I20260619 06:33:48.367272 31426 raft_consensus.cc:2229] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:33:48.367993 31426 raft_consensus.cc:2258] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:48.368741 31427 tablet_replica.cc:330] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a: stopping tablet replica
I20260619 06:33:48.369012 31428 tablet_replica.cc:330] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d: stopping tablet replica
I20260619 06:33:48.369056 31427 raft_consensus.cc:2229] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:48.369298 31428 raft_consensus.cc:2229] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:48.369410 31427 raft_consensus.cc:2258] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:48.369611 31428 raft_consensus.cc:2258] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:48.369608 31426 ts_tablet_manager.cc:1905] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:48.370523 31427 ts_tablet_manager.cc:1905] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:48.370661 31428 ts_tablet_manager.cc:1905] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:48.376502 31426 ts_tablet_manager.cc:1918] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:33:48.376732 31426 log.cc:1196] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/7492bd636eee40559d8753a55a6ccc01
I20260619 06:33:48.377296 31426 ts_tablet_manager.cc:1939] T 7492bd636eee40559d8753a55a6ccc01 P fd77a969ff064b849ebd584641a116f1: Deleting consensus metadata
I20260619 06:33:48.377547 31428 ts_tablet_manager.cc:1918] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:33:48.377772 31428 log.cc:1196] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/7492bd636eee40559d8753a55a6ccc01
I20260619 06:33:48.378166 31427 ts_tablet_manager.cc:1918] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:33:48.378330 31428 ts_tablet_manager.cc:1939] T 7492bd636eee40559d8753a55a6ccc01 P eaeac2cee4e14496964e89b1714e796d: Deleting consensus metadata
I20260619 06:33:48.378372 31427 log.cc:1196] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/7492bd636eee40559d8753a55a6ccc01
I20260619 06:33:48.378816 30822 catalog_manager.cc:4841] TS fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013): tablet 7492bd636eee40559d8753a55a6ccc01 (table test-table [id=a7e8ece2972749699444a891282566d8]) successfully deleted
I20260619 06:33:48.379027 31427 ts_tablet_manager.cc:1939] T 7492bd636eee40559d8753a55a6ccc01 P d460e8fae9b045808279435753c9017a: Deleting consensus metadata
I20260619 06:33:48.379550 30822 catalog_manager.cc:4841] TS eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961): tablet 7492bd636eee40559d8753a55a6ccc01 (table test-table [id=a7e8ece2972749699444a891282566d8]) successfully deleted
I20260619 06:33:48.380214 30824 catalog_manager.cc:4841] TS d460e8fae9b045808279435753c9017a (127.28.104.194:40589): tablet 7492bd636eee40559d8753a55a6ccc01 (table test-table [id=a7e8ece2972749699444a891282566d8]) successfully deleted
Jun 19 06:33:48 dist-test-slave-nf4n krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:33:48 dist-test-slave-nf4n krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:48.419080 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.400343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.28.104.254:32843)
0619 06:33:48.400877 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:48.400882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:48.401434 (+ 552us) server_negotiation.cc:407] Connection header received
0619 06:33:48.401584 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:48.401588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:48.401653 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:48.401751 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:48.403188 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.404000 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.405287 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.405721 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.405826 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.414094 (+ 8268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:48.414099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:48.414103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:48.414143 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:48.416764 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:48.417478 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:48.417482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:48.417484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:48.417612 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:48.417925 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:48.417928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:48.417930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:48.418201 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:48.418386 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:48.418653 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 06:33:48.418820 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":280,"threads_started":1}
I20260619 06:33:48.422775 30837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:33:48.423430 30837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:33:48.430119 31112 tablet_service.cc:1467] Processing CreateTablet for tablet c945a2caa6ec4ad68647f5275e29ac2d (DEFAULT_TABLE table=test-table [id=bc158669909542c99effb7cac81e558b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:48.430146 30974 tablet_service.cc:1467] Processing CreateTablet for tablet c945a2caa6ec4ad68647f5275e29ac2d (DEFAULT_TABLE table=test-table [id=bc158669909542c99effb7cac81e558b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:48.430339 31252 tablet_service.cc:1467] Processing CreateTablet for tablet c945a2caa6ec4ad68647f5275e29ac2d (DEFAULT_TABLE table=test-table [id=bc158669909542c99effb7cac81e558b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:48.430573 30974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c945a2caa6ec4ad68647f5275e29ac2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:48.430622 31112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c945a2caa6ec4ad68647f5275e29ac2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:48.430816 31252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c945a2caa6ec4ad68647f5275e29ac2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:48.437582 31438 tablet_bootstrap.cc:492] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1: Bootstrap starting.
I20260619 06:33:48.438030 31437 tablet_bootstrap.cc:492] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d: Bootstrap starting.
I20260619 06:33:48.439852 31438 tablet_bootstrap.cc:654] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:48.440374 31437 tablet_bootstrap.cc:654] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:48.441227 31439 tablet_bootstrap.cc:492] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a: Bootstrap starting.
I20260619 06:33:48.442674 31437 tablet_bootstrap.cc:492] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d: No bootstrap required, opened a new log
I20260619 06:33:48.442682 31438 tablet_bootstrap.cc:492] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1: No bootstrap required, opened a new log
I20260619 06:33:48.442745 31439 tablet_bootstrap.cc:654] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:48.442945 31437 ts_tablet_manager.cc:1397] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:33:48.443231 31438 ts_tablet_manager.cc:1397] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 06:33:48.443925 31437 raft_consensus.cc:348] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.444017 31438 raft_consensus.cc:348] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.444219 31437 raft_consensus.cc:374] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:48.444355 31438 raft_consensus.cc:374] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:48.444383 31437 raft_consensus.cc:729] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaeac2cee4e14496964e89b1714e796d, State: Initialized, Role: FOLLOWER
I20260619 06:33:48.444463 31438 raft_consensus.cc:729] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd77a969ff064b849ebd584641a116f1, State: Initialized, Role: FOLLOWER
I20260619 06:33:48.444715 31438 consensus_queue.cc:260] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.444698 31437 consensus_queue.cc:260] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.445288 31438 ts_tablet_manager.cc:1428] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 06:33:48.445443 31439 tablet_bootstrap.cc:492] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a: No bootstrap required, opened a new log
I20260619 06:33:48.445461 31437 ts_tablet_manager.cc:1428] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:33:48.445647 31439 ts_tablet_manager.cc:1397] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:33:48.446410 31439 raft_consensus.cc:348] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.446645 31439 raft_consensus.cc:374] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:48.446749 31439 raft_consensus.cc:729] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d460e8fae9b045808279435753c9017a, State: Initialized, Role: FOLLOWER
I20260619 06:33:48.446985 31439 consensus_queue.cc:260] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.448638 31439 ts_tablet_manager.cc:1428] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:48.514376 31443 raft_consensus.cc:482] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:33:48.514628 31443 raft_consensus.cc:504] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.516219 31443 leader_election.cc:288] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013), eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961)
I20260619 06:33:48.524042 31445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.516578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:44941 (local address 127.28.104.193:44013)
0619 06:33:48.517786 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:48.517793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:48.517822 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:33:48.517907 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:48.517912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:48.517994 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:48.518844 (+ 850us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:48.519070 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.519914 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.522159 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.522922 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.523052 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.523104 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:48.523563 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 06:33:48.523704 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":1026,"threads_started":1}
I20260619 06:33:48.524097 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.517431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:49149 (local address 127.28.104.195:34961)
0619 06:33:48.518295 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:48.518300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:48.518319 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:33:48.518375 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:48.518378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:48.518431 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:48.518512 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:33:48.519211 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.519966 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.522110 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.523046 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.523245 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.523329 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:48.523627 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 06:33:48.523771 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":619,"threads_started":1}
I20260619 06:33:48.524135 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.517328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:34961 (local address 127.28.104.194:49149)
0619 06:33:48.518097 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:48.518118 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:48.518221 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:48.518580 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:48.518586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:48.518616 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:48.519057 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:48.519069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.520098 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.520107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:48.521890 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:48.521903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.523279 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.523283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:48.523437 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.523460 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:33:48.523571 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 06:33:48.523704 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":24,"thread_start_us":379,"threads_started":1}
I20260619 06:33:48.524173 31444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.516445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:44013 (local address 127.28.104.194:44941)
0619 06:33:48.517247 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:48.517273 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:48.517447 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:48.518182 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:48.518187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:48.518238 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:33:48.518622 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:48.518636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.520085 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.520096 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:48.521957 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:48.521975 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.523148 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.523156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:48.523382 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.523426 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 06:33:48.523508 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:33:48.523663 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":57,"spinlock_wait_cycles":7040,"thread_start_us":354,"threads_started":1}
I20260619 06:33:48.524860 31277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c945a2caa6ec4ad68647f5275e29ac2d" candidate_uuid: "d460e8fae9b045808279435753c9017a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaeac2cee4e14496964e89b1714e796d" is_pre_election: true
I20260619 06:33:48.525156 31277 raft_consensus.cc:2454] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d460e8fae9b045808279435753c9017a in term 0.
I20260619 06:33:48.525156 30997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c945a2caa6ec4ad68647f5275e29ac2d" candidate_uuid: "d460e8fae9b045808279435753c9017a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd77a969ff064b849ebd584641a116f1" is_pre_election: true
I20260619 06:33:48.525415 30997 raft_consensus.cc:2454] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d460e8fae9b045808279435753c9017a in term 0.
I20260619 06:33:48.525703 31065 leader_election.cc:302] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d460e8fae9b045808279435753c9017a, eaeac2cee4e14496964e89b1714e796d; no voters:
I20260619 06:33:48.526060 31443 raft_consensus.cc:2781] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:33:48.526194 31443 raft_consensus.cc:482] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:33:48.526294 31443 raft_consensus.cc:3037] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:48.528519 31443 raft_consensus.cc:504] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.529141 31443 leader_election.cc:288] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [CANDIDATE]: Term 1 election: Requested vote from peers fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013), eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961)
I20260619 06:33:48.529563 30997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c945a2caa6ec4ad68647f5275e29ac2d" candidate_uuid: "d460e8fae9b045808279435753c9017a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd77a969ff064b849ebd584641a116f1"
I20260619 06:33:48.529747 30997 raft_consensus.cc:3037] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:48.529690 31277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c945a2caa6ec4ad68647f5275e29ac2d" candidate_uuid: "d460e8fae9b045808279435753c9017a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaeac2cee4e14496964e89b1714e796d"
I20260619 06:33:48.529909 31277 raft_consensus.cc:3037] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:48.532367 30997 raft_consensus.cc:2454] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d460e8fae9b045808279435753c9017a in term 1.
I20260619 06:33:48.532567 31277 raft_consensus.cc:2454] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d460e8fae9b045808279435753c9017a in term 1.
I20260619 06:33:48.532775 31065 leader_election.cc:302] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d460e8fae9b045808279435753c9017a, fd77a969ff064b849ebd584641a116f1; no voters:
I20260619 06:33:48.533109 31443 raft_consensus.cc:2781] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:48.533540 31443 raft_consensus.cc:686] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 1 LEADER]: Becoming Leader. State: Replica: d460e8fae9b045808279435753c9017a, State: Running, Role: LEADER
I20260619 06:33:48.533946 31443 consensus_queue.cc:237] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.537467 30835 catalog_manager.cc:5495] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a reported cstate change: term changed from 0 to 1, leader changed from <none> to d460e8fae9b045808279435753c9017a (127.28.104.194). New cstate: current_term: 1 leader_uuid: "d460e8fae9b045808279435753c9017a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } health_report { overall_health: UNKNOWN } } }
I20260619 06:33:48.575058 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.568969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.28.104.194:40589)
0619 06:33:48.569727 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:48.569732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:48.569761 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:33:48.569873 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:48.569882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:48.570018 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:48.570226 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:48.570572 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.571427 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.572820 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.573330 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.573423 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.573868 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:48.574484 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:48.574597 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:48.574735 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 06:33:48.574796 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":396,"threads_started":1}
I20260619 06:33:48.590740 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.585276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:60561 (local address 127.28.104.195:34961)
0619 06:33:48.585518 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:48.585525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:48.585876 (+ 351us) server_negotiation.cc:407] Connection header received
0619 06:33:48.586100 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:48.586105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:48.586169 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:48.586266 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:48.586762 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.587469 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.588972 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.589438 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.589581 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.590004 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:48.590117 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:48.590207 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:48.590354 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 06:33:48.590437 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:33:48.590870 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.585000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:34961 (local address 127.28.104.194:60561)
0619 06:33:48.585774 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:48.585790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:48.585955 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:48.586280 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:48.586285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:48.586296 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:33:48.586606 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:48.586615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.587635 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.587642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:48.588806 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:48.588818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.589620 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.589630 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:48.589772 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.589825 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:33:48.590223 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:33:48.590237 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 06:33:48.590330 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:33:48.590469 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":402,"threads_started":1}
I20260619 06:33:48.599334 31277 raft_consensus.cc:1261] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Refusing update from remote peer d460e8fae9b045808279435753c9017a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:48.599380 30997 raft_consensus.cc:1261] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Refusing update from remote peer d460e8fae9b045808279435753c9017a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:48.599993 31448 consensus_queue.cc:1035] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:48.600389 31443 consensus_queue.cc:1035] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:48.628222 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.623034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:35963 (local address 127.28.104.194:40589)
0619 06:33:48.623183 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:48.623189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:48.623448 (+ 259us) server_negotiation.cc:407] Connection header received
0619 06:33:48.623565 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:48.623568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:48.623627 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:48.623719 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:48.624080 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.624929 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.626530 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.626971 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.627112 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.627506 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:48.627615 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:48.627713 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:48.627860 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 06:33:48.627935 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:33:48.628319 31459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.622734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40589 (local address 127.28.104.195:35963)
0619 06:33:48.623360 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:48.623376 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:48.623481 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:48.623742 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:48.623745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:48.623754 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:33:48.623971 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:48.623978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.625111 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.625118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:48.626385 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:48.626396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.627141 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.627148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:48.627309 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.627346 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:33:48.627726 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:33:48.627740 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 06:33:48.627837 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:33:48.627961 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":355,"threads_started":1}
I20260619 06:33:48.660068 30835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:48.660385 30835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:48.663480 30835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5: Sending DeleteTablet for 3 replicas of tablet c945a2caa6ec4ad68647f5275e29ac2d
I20260619 06:33:48.664362 30974 tablet_service.cc:1514] Processing DeleteTablet for tablet c945a2caa6ec4ad68647f5275e29ac2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:48 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:37354
I20260619 06:33:48.664386 31112 tablet_service.cc:1514] Processing DeleteTablet for tablet c945a2caa6ec4ad68647f5275e29ac2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:48 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:58262
I20260619 06:33:48.664412 31252 tablet_service.cc:1514] Processing DeleteTablet for tablet c945a2caa6ec4ad68647f5275e29ac2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:48 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:34696
I20260619 06:33:48.664659 31426 tablet_replica.cc:330] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1: stopping tablet replica
I20260619 06:33:48.664666 31427 tablet_replica.cc:330] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a: stopping tablet replica
I20260619 06:33:48.664862 31427 raft_consensus.cc:2229] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:33:48.664860 31426 raft_consensus.cc:2229] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:48.665112 31426 raft_consensus.cc:2258] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:48.665176 31428 tablet_replica.cc:330] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d: stopping tablet replica
I20260619 06:33:48.665380 31428 raft_consensus.cc:2229] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:48.665557 31428 raft_consensus.cc:2258] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:48.666257 31426 ts_tablet_manager.cc:1905] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:48.666356 31427 raft_consensus.cc:2258] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:48.666249 30835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:33:48.666894 30835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:33:48.667523 31427 ts_tablet_manager.cc:1905] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:48.670253 31428 ts_tablet_manager.cc:1905] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:48.672842 31426 ts_tablet_manager.cc:1918] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:33:48.672981 31426 log.cc:1196] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/c945a2caa6ec4ad68647f5275e29ac2d
I20260619 06:33:48.673460 31426 ts_tablet_manager.cc:1939] T c945a2caa6ec4ad68647f5275e29ac2d P fd77a969ff064b849ebd584641a116f1: Deleting consensus metadata
I20260619 06:33:48.674984 31427 ts_tablet_manager.cc:1918] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:33:48.675237 31427 log.cc:1196] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/c945a2caa6ec4ad68647f5275e29ac2d
I20260619 06:33:48.675577 30822 catalog_manager.cc:4841] TS fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013): tablet c945a2caa6ec4ad68647f5275e29ac2d (table test-table [id=bc158669909542c99effb7cac81e558b]) successfully deleted
I20260619 06:33:48.675729 31427 ts_tablet_manager.cc:1939] T c945a2caa6ec4ad68647f5275e29ac2d P d460e8fae9b045808279435753c9017a: Deleting consensus metadata
I20260619 06:33:48.676865 30824 catalog_manager.cc:4841] TS d460e8fae9b045808279435753c9017a (127.28.104.194:40589): tablet c945a2caa6ec4ad68647f5275e29ac2d (table test-table [id=bc158669909542c99effb7cac81e558b]) successfully deleted
I20260619 06:33:48.677346 31428 ts_tablet_manager.cc:1918] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:33:48.677510 31428 log.cc:1196] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/c945a2caa6ec4ad68647f5275e29ac2d
I20260619 06:33:48.677928 31428 ts_tablet_manager.cc:1939] T c945a2caa6ec4ad68647f5275e29ac2d P eaeac2cee4e14496964e89b1714e796d: Deleting consensus metadata
I20260619 06:33:48.678797 30822 catalog_manager.cc:4841] TS eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961): tablet c945a2caa6ec4ad68647f5275e29ac2d (table test-table [id=bc158669909542c99effb7cac81e558b]) successfully deleted
I20260619 06:33:48.683007 30974 tablet_service.cc:1467] Processing CreateTablet for tablet 79a0031b262a46be91450d9b295617ca (DEFAULT_TABLE table=test-table [id=4c37bfad043c425e92a6690811f43cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:48.683350 31252 tablet_service.cc:1467] Processing CreateTablet for tablet 79a0031b262a46be91450d9b295617ca (DEFAULT_TABLE table=test-table [id=4c37bfad043c425e92a6690811f43cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:48.683480 30974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a0031b262a46be91450d9b295617ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:48.683462 31112 tablet_service.cc:1467] Processing CreateTablet for tablet 79a0031b262a46be91450d9b295617ca (DEFAULT_TABLE table=test-table [id=4c37bfad043c425e92a6690811f43cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:33:48.683739 31252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a0031b262a46be91450d9b295617ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:48.683849 31112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a0031b262a46be91450d9b295617ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:48.690224 31438 tablet_bootstrap.cc:492] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1: Bootstrap starting.
I20260619 06:33:48.690387 31437 tablet_bootstrap.cc:492] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d: Bootstrap starting.
I20260619 06:33:48.691926 31437 tablet_bootstrap.cc:654] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:48.691932 31438 tablet_bootstrap.cc:654] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:48.694554 31437 tablet_bootstrap.cc:492] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d: No bootstrap required, opened a new log
I20260619 06:33:48.694736 31437 ts_tablet_manager.cc:1397] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 06:33:48.694792 31438 tablet_bootstrap.cc:492] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1: No bootstrap required, opened a new log
I20260619 06:33:48.694984 31438 ts_tablet_manager.cc:1397] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260619 06:33:48.695533 31437 raft_consensus.cc:348] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.695703 31438 raft_consensus.cc:348] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.695935 31438 raft_consensus.cc:374] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:48.696024 31438 raft_consensus.cc:729] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd77a969ff064b849ebd584641a116f1, State: Initialized, Role: FOLLOWER
I20260619 06:33:48.696153 31437 raft_consensus.cc:374] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:48.696244 31437 raft_consensus.cc:729] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaeac2cee4e14496964e89b1714e796d, State: Initialized, Role: FOLLOWER
I20260619 06:33:48.696493 31438 consensus_queue.cc:260] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.696969 31437 consensus_queue.cc:260] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.698470 31438 ts_tablet_manager.cc:1428] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 06:33:48.697513 31437 ts_tablet_manager.cc:1428] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:33:48.699424 31439 tablet_bootstrap.cc:492] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a: Bootstrap starting.
I20260619 06:33:48.701272 31439 tablet_bootstrap.cc:654] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:48.704815 31439 tablet_bootstrap.cc:492] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a: No bootstrap required, opened a new log
I20260619 06:33:48.705013 31439 ts_tablet_manager.cc:1397] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260619 06:33:48.705801 31439 raft_consensus.cc:348] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.706248 31439 raft_consensus.cc:374] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:48.706398 31439 raft_consensus.cc:729] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d460e8fae9b045808279435753c9017a, State: Initialized, Role: FOLLOWER
I20260619 06:33:48.706707 31439 consensus_queue.cc:260] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.707294 31439 ts_tablet_manager.cc:1428] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:33:48.863440 31373 raft_consensus.cc:482] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:33:48.863698 31373 raft_consensus.cc:504] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.864363 31373 leader_election.cc:288] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d460e8fae9b045808279435753c9017a (127.28.104.194:40589), eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961)
I20260619 06:33:48.864990 31277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a0031b262a46be91450d9b295617ca" candidate_uuid: "fd77a969ff064b849ebd584641a116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaeac2cee4e14496964e89b1714e796d" is_pre_election: true
I20260619 06:33:48.864979 31132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a0031b262a46be91450d9b295617ca" candidate_uuid: "fd77a969ff064b849ebd584641a116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d460e8fae9b045808279435753c9017a" is_pre_election: true
I20260619 06:33:48.865289 31277 raft_consensus.cc:2454] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd77a969ff064b849ebd584641a116f1 in term 0.
I20260619 06:33:48.865314 31132 raft_consensus.cc:2454] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd77a969ff064b849ebd584641a116f1 in term 0.
I20260619 06:33:48.865787 30925 leader_election.cc:302] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaeac2cee4e14496964e89b1714e796d, fd77a969ff064b849ebd584641a116f1; no voters:
I20260619 06:33:48.866101 31373 raft_consensus.cc:2781] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:33:48.866209 31373 raft_consensus.cc:482] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:33:48.866299 31373 raft_consensus.cc:3037] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:48.869186 31373 raft_consensus.cc:504] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.869820 31373 leader_election.cc:288] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [CANDIDATE]: Term 1 election: Requested vote from peers d460e8fae9b045808279435753c9017a (127.28.104.194:40589), eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961)
I20260619 06:33:48.870332 31132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a0031b262a46be91450d9b295617ca" candidate_uuid: "fd77a969ff064b849ebd584641a116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d460e8fae9b045808279435753c9017a"
I20260619 06:33:48.870354 31277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a0031b262a46be91450d9b295617ca" candidate_uuid: "fd77a969ff064b849ebd584641a116f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaeac2cee4e14496964e89b1714e796d"
I20260619 06:33:48.870571 31277 raft_consensus.cc:3037] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:48.870577 31132 raft_consensus.cc:3037] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:48.873553 31277 raft_consensus.cc:2454] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd77a969ff064b849ebd584641a116f1 in term 1.
I20260619 06:33:48.873939 31132 raft_consensus.cc:2454] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd77a969ff064b849ebd584641a116f1 in term 1.
I20260619 06:33:48.874089 30925 leader_election.cc:302] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaeac2cee4e14496964e89b1714e796d, fd77a969ff064b849ebd584641a116f1; no voters:
I20260619 06:33:48.874410 31373 raft_consensus.cc:2781] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:48.874990 31373 raft_consensus.cc:686] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 1 LEADER]: Becoming Leader. State: Replica: fd77a969ff064b849ebd584641a116f1, State: Running, Role: LEADER
I20260619 06:33:48.875355 31373 consensus_queue.cc:237] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } }
I20260619 06:33:48.878674 30835 catalog_manager.cc:5495] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd77a969ff064b849ebd584641a116f1 (127.28.104.193). New cstate: current_term: 1 leader_uuid: "fd77a969ff064b849ebd584641a116f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd77a969ff064b849ebd584641a116f1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 44013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 } health_report { overall_health: UNKNOWN } } }
I20260619 06:33:48.909989 31445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:48.904997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.28.104.193:44013)
0619 06:33:48.905204 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:48.905211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:48.905421 (+ 210us) server_negotiation.cc:407] Connection header received
0619 06:33:48.905640 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:48.905646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:48.905720 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:48.905818 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:48.906198 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.906921 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.908268 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:48.908684 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:48.908819 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:48.909339 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:48.909436 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:48.909534 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:48.909653 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 06:33:48.909728 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:33:48.920231 31132 raft_consensus.cc:1261] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Refusing update from remote peer fd77a969ff064b849ebd584641a116f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:48.920912 31464 consensus_queue.cc:1035] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d460e8fae9b045808279435753c9017a" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:48.921469 31277 raft_consensus.cc:1261] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Refusing update from remote peer fd77a969ff064b849ebd584641a116f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:48.922853 31464 consensus_queue.cc:1035] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaeac2cee4e14496964e89b1714e796d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260619 06:33:48.955665 31409 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:33:50.544436 30837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:50.544704 30837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:33:50.548799 30837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44de544ccc2f4adaa4bf98a77a72f3f5: Sending DeleteTablet for 3 replicas of tablet 79a0031b262a46be91450d9b295617ca
I20260619 06:33:50.549759 31112 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a0031b262a46be91450d9b295617ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:50 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:58262
I20260619 06:33:50.549800 31252 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a0031b262a46be91450d9b295617ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:50 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:34696
I20260619 06:33:50.549836 30974 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a0031b262a46be91450d9b295617ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:33:50 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:37354
I20260619 06:33:50.550405 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 30908
I20260619 06:33:50.572878 31493 tablet_replica.cc:330] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a: stopping tablet replica
I20260619 06:33:50.573247 31493 raft_consensus.cc:2229] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:50.573539 31493 raft_consensus.cc:2258] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:50.574117 31492 tablet_replica.cc:330] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d: stopping tablet replica
I20260619 06:33:50.574352 31492 raft_consensus.cc:2229] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:33:50.574599 31492 raft_consensus.cc:2258] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:50.575137 31493 ts_tablet_manager.cc:1905] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:50.575151 31494 tablet_replica.cc:330] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1: stopping tablet replica
I20260619 06:33:50.575372 31494 raft_consensus.cc:2229] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:33:50.575843 31494 raft_consensus.cc:2258] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:33:50.576673 31492 ts_tablet_manager.cc:1905] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:50.576930 31494 ts_tablet_manager.cc:1905] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:33:50.582132 31493 ts_tablet_manager.cc:1918] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:33:50.582360 31493 log.cc:1196] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/79a0031b262a46be91450d9b295617ca
Jun 19 06:33:50 dist-test-slave-nf4n krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:33:50.582868 31493 ts_tablet_manager.cc:1939] T 79a0031b262a46be91450d9b295617ca P d460e8fae9b045808279435753c9017a: Deleting consensus metadata
I20260619 06:33:50.584002 31492 ts_tablet_manager.cc:1918] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:33:50.584152 31492 log.cc:1196] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/79a0031b262a46be91450d9b295617ca
I20260619 06:33:50.584126 30824 catalog_manager.cc:4841] TS d460e8fae9b045808279435753c9017a (127.28.104.194:40589): tablet 79a0031b262a46be91450d9b295617ca (table test-table [id=4c37bfad043c425e92a6690811f43cfb]) successfully deleted
I20260619 06:33:50.584447 31494 ts_tablet_manager.cc:1918] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:33:50.584597 31494 log.cc:1196] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/79a0031b262a46be91450d9b295617ca
I20260619 06:33:50.584614 31492 ts_tablet_manager.cc:1939] T 79a0031b262a46be91450d9b295617ca P eaeac2cee4e14496964e89b1714e796d: Deleting consensus metadata
I20260619 06:33:50.585089 31494 ts_tablet_manager.cc:1939] T 79a0031b262a46be91450d9b295617ca P fd77a969ff064b849ebd584641a116f1: Deleting consensus metadata
I20260619 06:33:50.585804 30822 catalog_manager.cc:4841] TS eaeac2cee4e14496964e89b1714e796d (127.28.104.195:34961): tablet 79a0031b262a46be91450d9b295617ca (table test-table [id=4c37bfad043c425e92a6690811f43cfb]) successfully deleted
I20260619 06:33:50.586246 30822 catalog_manager.cc:4841] TS fd77a969ff064b849ebd584641a116f1 (127.28.104.193:44013): tablet 79a0031b262a46be91450d9b295617ca (table test-table [id=4c37bfad043c425e92a6690811f43cfb]) successfully deleted
I20260619 06:33:50.590516 31495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:50.551745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.28.104.193:44013)
0619 06:33:50.571944 (+ 20199us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:50.571953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:50.571987 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:33:50.572072 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:50.572081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:50.572189 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:50.572314 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:50.574152 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:50.575398 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:50.577386 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:50.578031 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:50.578137 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:50.584546 (+ 6409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:50.584569 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:50.584598 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:50.584651 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:50.587895 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:50.588588 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:50.588595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:50.588598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:50.588739 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:50.589121 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:50.589124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:50.589126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:50.589489 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:50.589707 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:50.589953 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 06:33:50.590133 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19933,"spinlock_wait_cycles":6912,"thread_start_us":19922,"threads_started":1}
W20260619 06:33:50.591234 31039 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37376
W20260619 06:33:50.591858 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 30908: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:33:50.592067 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 30908
I20260619 06:33:50.615321 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31048
Jun 19 06:33:50 dist-test-slave-nf4n krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:33:50.630889 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:50.616095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.28.104.194:40589)
0619 06:33:50.616680 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:50.616685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:50.616704 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:33:50.616765 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:50.616769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:50.616843 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:50.616949 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:50.618269 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:50.619005 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:50.620208 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:50.620616 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:50.620707 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:50.624801 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:50.624817 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:50.624832 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:50.624868 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:50.628129 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:50.628781 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:50.628788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:50.628791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:50.628969 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:50.629436 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:50.629448 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:50.629459 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:50.629901 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:50.630261 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:50.630393 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 06:33:50.630595 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":318,"threads_started":1}
W20260619 06:33:50.631721 31178 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58286
W20260619 06:33:50.632408 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31048: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:33:50.632563 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 31048
I20260619 06:33:50.655519 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31187
Jun 19 06:33:50 dist-test-slave-nf4n krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:33:50.671536 31498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:50.656223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.28.104.195:34961)
0619 06:33:50.656750 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:50.656754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:50.656769 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:33:50.656822 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:50.656825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:50.656887 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:50.657009 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:50.658343 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:50.659171 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:50.660421 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:50.660864 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:50.660993 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:50.665565 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:50.665587 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:50.665604 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:50.665642 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:50.668491 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:50.669387 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:50.669404 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:50.669407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:50.669535 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:50.669951 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:50.669960 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:50.669962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:50.670322 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:50.670631 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:50.670943 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 06:33:50.671197 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":245,"threads_started":1}
W20260619 06:33:50.672399 31317 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34720
W20260619 06:33:50.673156 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31187: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:33:50.673331 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 31187
I20260619 06:33:50.698546 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 30805
I20260619 06:33:50.699728 30877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:50.820472 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 30805
2026-06-19T06:33:50Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8429 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:33:50 dist-test-slave-nf4n krb5kdc[31502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:33:50 dist-test-slave-nf4n krb5kdc[31502](info): set up 2 sockets
Jun 19 06:33:50 dist-test-slave-nf4n krb5kdc[31502](info): commencing operation
krb5kdc: starting...
W20260619 06:33:51.921083 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:33:52 dist-test-slave-nf4n krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:33:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:33:52.112680 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:34261
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:33099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:34261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:52.434060 31515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:52.434360 31515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:52.434525 31515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:52.441946 31515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:52.442050 31515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:52.442157 31515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:52.442250 31515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:52.442333 31515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:52.452309 31515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:33099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:34261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:34261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:52.453697 31515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:52.455466 31515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:52.468199 31520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:52.468174 31521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:52.468467 31523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:52.468673 31515 server_base.cc:1034] running on GCE node
I20260619 06:33:52.469578 31515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:52.470935 31515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:52.472150 31515 hybrid_clock.cc:648] HybridClock initialized: now 1781850832472106 us; error 31 us; skew 500 ppm
I20260619 06:33:52.472450 31515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:52 dist-test-slave-nf4n krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850832, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:52.475709 31515 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:33:52.477239 31515 webserver.cc:460] Webserver started at http://127.28.104.254:41135/ using document root <none> and password file <none>
I20260619 06:33:52.477941 31515 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:52.478068 31515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:52.478404 31515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:52.481482 31515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "d0dfa18246b54bb9a575a7c216d199bf"
format_stamp: "Formatted at 2026-06-19 06:33:52 on dist-test-slave-nf4n"
server_key: "70e5140520e3c8dfda9a826168796c3a"
server_key_iv: "340c2d0b4f4ec8ad3af841e58cd31341"
server_key_version: "encryptionkey@0"
I20260619 06:33:52.482259 31515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "d0dfa18246b54bb9a575a7c216d199bf"
format_stamp: "Formatted at 2026-06-19 06:33:52 on dist-test-slave-nf4n"
server_key: "70e5140520e3c8dfda9a826168796c3a"
server_key_iv: "340c2d0b4f4ec8ad3af841e58cd31341"
server_key_version: "encryptionkey@0"
I20260619 06:33:52.487623 31515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 06:33:52.491374 31530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:52.492914 31515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:33:52.493203 31515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "d0dfa18246b54bb9a575a7c216d199bf"
format_stamp: "Formatted at 2026-06-19 06:33:52 on dist-test-slave-nf4n"
server_key: "70e5140520e3c8dfda9a826168796c3a"
server_key_iv: "340c2d0b4f4ec8ad3af841e58cd31341"
server_key_version: "encryptionkey@0"
I20260619 06:33:52.493448 31515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:52.507782 31515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:52.512521 31515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:52.512898 31515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:52.534291 31515 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:34261
I20260619 06:33:52.534258 31581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:34261 every 8 connection(s)
I20260619 06:33:52.535957 31515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:33:52.540048 31582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:52.540589 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 31515
I20260619 06:33:52.540800 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:33:52.541507 29091 external_mini_cluster.cc:1428] Setting key 5acf3e2f0ac9e2f5f0b0a84b42534610
I20260619 06:33:52.554456 31582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf: Bootstrap starting.
Jun 19 06:33:52 dist-test-slave-nf4n krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:52.558437 31582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:52.559542 31582 log.cc:824] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:52.562947 31582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf: No bootstrap required, opened a new log
I20260619 06:33:52.566751 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:52.543478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.28.104.254:34261)
0619 06:33:52.544256 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:52.544267 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:52.544308 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:33:52.546306 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:52.546336 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:52.546769 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:52.547134 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:52.548966 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:52.550467 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:52.552337 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:52.553201 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:52.553368 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:52.558208 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:52.558222 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:52.558237 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:52.558275 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:52.562012 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:52.562742 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:52.562750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:52.562758 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:52.562981 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:52.563432 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:52.563438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:52.563442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:52.564154 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:52.564458 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:52.564812 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 06:33:52.565311 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":359,"threads_started":1}
I20260619 06:33:52.571848 31582 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dfa18246b54bb9a575a7c216d199bf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34261 } }
I20260619 06:33:52.572224 31582 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:52.572355 31582 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dfa18246b54bb9a575a7c216d199bf, State: Initialized, Role: FOLLOWER
I20260619 06:33:52.572983 31582 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dfa18246b54bb9a575a7c216d199bf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34261 } }
I20260619 06:33:52.573274 31582 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:33:52.573401 31582 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:33:52.573581 31582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:52.575713 31582 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dfa18246b54bb9a575a7c216d199bf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34261 } }
I20260619 06:33:52.576238 31582 leader_election.cc:302] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0dfa18246b54bb9a575a7c216d199bf; no voters:
I20260619 06:33:52.576793 31582 leader_election.cc:288] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:33:52.576934 31587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:52.578079 31587 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [term 1 LEADER]: Becoming Leader. State: Replica: d0dfa18246b54bb9a575a7c216d199bf, State: Running, Role: LEADER
I20260619 06:33:52.578583 31587 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dfa18246b54bb9a575a7c216d199bf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34261 } }
I20260619 06:33:52.579069 31582 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:52.581357 31588 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0dfa18246b54bb9a575a7c216d199bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dfa18246b54bb9a575a7c216d199bf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34261 } } }
I20260619 06:33:52.581780 31588 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:52.581697 31589 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0dfa18246b54bb9a575a7c216d199bf. Latest consensus state: current_term: 1 leader_uuid: "d0dfa18246b54bb9a575a7c216d199bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dfa18246b54bb9a575a7c216d199bf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34261 } } }
I20260619 06:33:52.581967 31589 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:52.582600 31595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:33:52.587762 31595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:33:52.596705 31595 catalog_manager.cc:1347] Generated new cluster ID: 2e912bfbccda43eb9fcd0191e041f93d
I20260619 06:33:52.596835 31595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:33:52.608306 31595 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:33:52.609742 31595 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:33:52.617277 31595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0dfa18246b54bb9a575a7c216d199bf: Generated new TSK 0
I20260619 06:33:52.618083 31595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:33:52.669386 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34261
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:33099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:53.003365 31608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:53.003679 31608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:53.003852 31608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:53.011147 31608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:53.011251 31608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:53.011430 31608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:33:53.021379 31608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:33099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:53.022689 31608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:53.024061 31608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:53.037757 31614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:53.038266 31613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:53.037720 31616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:53.039093 31608 server_base.cc:1034] running on GCE node
I20260619 06:33:53.039804 31608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:53.040764 31608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:53.041989 31608 hybrid_clock.cc:648] HybridClock initialized: now 1781850833041918 us; error 63 us; skew 500 ppm
I20260619 06:33:53.042348 31608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:53 dist-test-slave-nf4n krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850833, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:53.045616 31608 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:33:53.047052 31608 webserver.cc:460] Webserver started at http://127.28.104.193:40279/ using document root <none> and password file <none>
I20260619 06:33:53.047712 31608 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:53.047804 31608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:53.048054 31608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:53.050973 31608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "3d189f87c41a464ba59dd661c2508552"
format_stamp: "Formatted at 2026-06-19 06:33:53 on dist-test-slave-nf4n"
server_key: "cdb7a15f547da5f3e560274652fa282e"
server_key_iv: "5406dc0f329bc1d7c50b5e4ceb5d7acc"
server_key_version: "encryptionkey@0"
I20260619 06:33:53.051702 31608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "3d189f87c41a464ba59dd661c2508552"
format_stamp: "Formatted at 2026-06-19 06:33:53 on dist-test-slave-nf4n"
server_key: "cdb7a15f547da5f3e560274652fa282e"
server_key_iv: "5406dc0f329bc1d7c50b5e4ceb5d7acc"
server_key_version: "encryptionkey@0"
I20260619 06:33:53.056943 31608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:33:53.060760 31623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:53.062377 31608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:53.062570 31608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "3d189f87c41a464ba59dd661c2508552"
format_stamp: "Formatted at 2026-06-19 06:33:53 on dist-test-slave-nf4n"
server_key: "cdb7a15f547da5f3e560274652fa282e"
server_key_iv: "5406dc0f329bc1d7c50b5e4ceb5d7acc"
server_key_version: "encryptionkey@0"
I20260619 06:33:53.062786 31608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:53.086611 31608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:53.091020 31608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:53.091356 31608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:53.092394 31608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:53.093924 31608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:53.094044 31608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:53.094211 31608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:53.094293 31608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:53.129701 31608 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:35107
I20260619 06:33:53.129743 31735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:35107 every 8 connection(s)
I20260619 06:33:53.131140 31608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:33:53.138351 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 31608
I20260619 06:33:53.138562 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:33:53.139356 29091 external_mini_cluster.cc:1428] Setting key e79d8b757e578fd9cf4a0d6c78d00204
Jun 19 06:33:53 dist-test-slave-nf4n krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850833, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:53.152328 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:53.133740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:58473 (local address 127.28.104.254:34261)
0619 06:33:53.134206 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:53.134211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:53.135108 (+ 897us) server_negotiation.cc:407] Connection header received
0619 06:33:53.136914 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:53.136921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:53.137013 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:53.137183 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:53.139159 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:53.140049 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:53.141946 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:53.142368 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:53.142468 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:53.146669 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:53.146674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:53.146677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:53.146709 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:53.149340 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:53.150042 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:53.150048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:53.150052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:53.150173 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:53.150588 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:53.150595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:53.150599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:53.150876 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:53.151027 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:53.151742 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 06:33:53.151968 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":247,"threads_started":1}
I20260619 06:33:53.153417 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:53.134258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34261 (local address 127.28.104.193:58473)
0619 06:33:53.134943 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:53.134988 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:53.136648 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:53.137424 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:53.137433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:53.138062 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:53.138942 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:53.138968 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:53.140279 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:53.140283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:53.141800 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:53.141817 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:53.142602 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:53.142613 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:53.142952 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:53.143717 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:53.143751 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:53.146437 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:53.149477 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:53.149485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:53.149503 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:53.149918 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:53.150283 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:53.150288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:53.150292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:53.150479 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:53.151002 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:53.151010 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:53.151368 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 06:33:53.151642 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 06:33:53.152199 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":260,"threads_started":1}
I20260619 06:33:53.155026 31736 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34261
I20260619 06:33:53.155377 31736 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:53.156299 31736 heartbeater.cc:511] Master 127.28.104.254:34261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
I20260619 06:33:53.158663 31547 ts_manager.cc:194] Registered new tserver with Master: 3d189f87c41a464ba59dd661c2508552 (127.28.104.193:35107)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
I20260619 06:33:53.160260 31547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:58473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:33:53.182668 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34261
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:33099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:53.504103 31742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:53.504380 31742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:53.504472 31742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:53.511869 31742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:53.511981 31742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:53.512094 31742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:33:53.521631 31742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:33099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:53.522864 31742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:53.524107 31742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:53.537807 31748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:53.538082 31747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:53.538306 31750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:53.538693 31742 server_base.cc:1034] running on GCE node
I20260619 06:33:53.539371 31742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:53.540462 31742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:53.541677 31742 hybrid_clock.cc:648] HybridClock initialized: now 1781850833541618 us; error 62 us; skew 500 ppm
I20260619 06:33:53.542024 31742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:53 dist-test-slave-nf4n krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850833, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:53.549120 31742 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:33:53.550704 31742 webserver.cc:460] Webserver started at http://127.28.104.194:37573/ using document root <none> and password file <none>
I20260619 06:33:53.551402 31742 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:53.551527 31742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:53.551851 31742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:53.554832 31742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "5e56c1dc8d6e4dba9d851bf69b0e1ac7"
format_stamp: "Formatted at 2026-06-19 06:33:53 on dist-test-slave-nf4n"
server_key: "cf2ae4887ae6e829506b0fdffe65a424"
server_key_iv: "071449633d8d9dfa805d2de194d7df4f"
server_key_version: "encryptionkey@0"
I20260619 06:33:53.555629 31742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "5e56c1dc8d6e4dba9d851bf69b0e1ac7"
format_stamp: "Formatted at 2026-06-19 06:33:53 on dist-test-slave-nf4n"
server_key: "cf2ae4887ae6e829506b0fdffe65a424"
server_key_iv: "071449633d8d9dfa805d2de194d7df4f"
server_key_version: "encryptionkey@0"
I20260619 06:33:53.560968 31742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 06:33:53.564532 31757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:53.565994 31742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:53.566169 31742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "5e56c1dc8d6e4dba9d851bf69b0e1ac7"
format_stamp: "Formatted at 2026-06-19 06:33:53 on dist-test-slave-nf4n"
server_key: "cf2ae4887ae6e829506b0fdffe65a424"
server_key_iv: "071449633d8d9dfa805d2de194d7df4f"
server_key_version: "encryptionkey@0"
I20260619 06:33:53.566385 31742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:53.583158 31742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:53.587404 31742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:53.587698 31742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:53.588685 31742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:53.590162 31742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:53.590269 31742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:53.590366 31742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:53.590409 31742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:53.623667 31742 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:36049
I20260619 06:33:53.623761 31869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:36049 every 8 connection(s)
I20260619 06:33:53.624993 31742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:33:53.631299 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 31742
I20260619 06:33:53.631479 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:33:53.632035 29091 external_mini_cluster.cc:1428] Setting key e500cea250ccc2037a4125f5d44f8e0e
Jun 19 06:33:53 dist-test-slave-nf4n krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850833, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:53.646625 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:53.627659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:46135 (local address 127.28.104.254:34261)
0619 06:33:53.627846 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:53.627851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:53.629095 (+ 1244us) server_negotiation.cc:407] Connection header received
0619 06:33:53.630954 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:53.630958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:53.631026 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:53.631113 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:53.633131 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:53.633968 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:53.635476 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:53.635902 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:53.636026 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:53.639430 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:53.639466 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:53.639474 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:53.639518 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:53.643187 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:53.643964 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:53.643972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:53.643978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:53.644137 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:53.644542 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:53.644550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:53.644555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:53.644943 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:53.645125 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:53.645907 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 06:33:53.646167 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:33:53.647431 31872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:53.628186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34261 (local address 127.28.104.194:46135)
0619 06:33:53.628899 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:53.628938 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:53.630678 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:53.631403 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:53.631415 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:53.632165 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:53.632936 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:53.632955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:53.634114 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:53.634124 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:53.635333 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:53.635344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:53.636126 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:53.636133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:53.636359 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:53.637100 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:53.637125 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:53.639206 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:53.643346 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:53.643354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:53.643367 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:53.643824 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:53.644244 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:53.644247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:53.644249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:53.644426 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:53.645169 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:53.645175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:53.645512 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 06:33:53.645759 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 06:33:53.646159 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":269,"threads_started":1}
I20260619 06:33:53.649005 31870 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34261
I20260619 06:33:53.649380 31870 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:53.650367 31870 heartbeater.cc:511] Master 127.28.104.254:34261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
I20260619 06:33:53.652206 31547 ts_manager.cc:194] Registered new tserver with Master: 5e56c1dc8d6e4dba9d851bf69b0e1ac7 (127.28.104.194:36049)
I20260619 06:33:53.653261 31547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:46135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:33:53.677711 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34261
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:33099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:54.001375 31875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:54.001735 31875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:54.001909 31875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:54.009224 31875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:54.009366 31875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:54.009554 31875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:33:54.019534 31875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:33099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:54.020907 31875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:54.022310 31875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:54.037058 31881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:54.037159 31880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:54.037156 31883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:54.038342 31875 server_base.cc:1034] running on GCE node
I20260619 06:33:54.039130 31875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:54.040186 31875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:54.041402 31875 hybrid_clock.cc:648] HybridClock initialized: now 1781850834041349 us; error 42 us; skew 500 ppm
I20260619 06:33:54.041759 31875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:54 dist-test-slave-nf4n krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:54.045435 31875 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:33:54.047112 31875 webserver.cc:460] Webserver started at http://127.28.104.195:39585/ using document root <none> and password file <none>
I20260619 06:33:54.047878 31875 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:54.047981 31875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:54.048338 31875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:54.051419 31875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "6ee290176374460a83665a5f2ddba649"
format_stamp: "Formatted at 2026-06-19 06:33:54 on dist-test-slave-nf4n"
server_key: "41967934e1723d02da9bbd34e2c4ca6e"
server_key_iv: "ee7b76243813c1469880faa81dc6484a"
server_key_version: "encryptionkey@0"
I20260619 06:33:54.052219 31875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "6ee290176374460a83665a5f2ddba649"
format_stamp: "Formatted at 2026-06-19 06:33:54 on dist-test-slave-nf4n"
server_key: "41967934e1723d02da9bbd34e2c4ca6e"
server_key_iv: "ee7b76243813c1469880faa81dc6484a"
server_key_version: "encryptionkey@0"
I20260619 06:33:54.057753 31875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:33:54.061674 31890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:54.063045 31875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:33:54.063217 31875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "6ee290176374460a83665a5f2ddba649"
format_stamp: "Formatted at 2026-06-19 06:33:54 on dist-test-slave-nf4n"
server_key: "41967934e1723d02da9bbd34e2c4ca6e"
server_key_iv: "ee7b76243813c1469880faa81dc6484a"
server_key_version: "encryptionkey@0"
I20260619 06:33:54.063441 31875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:54.076967 31875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:54.081771 31875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:54.082084 31875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:54.083101 31875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:54.084803 31875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:54.084918 31875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:54.085064 31875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:54.085134 31875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 06:33:54.118952 31875 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:33213
I20260619 06:33:54.119004 32002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:33213 every 8 connection(s)
I20260619 06:33:54.120242 31875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:33:54.126559 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 31875
I20260619 06:33:54.126757 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:33:54.127204 29091 external_mini_cluster.cc:1428] Setting key 6bbc531ecb581728f0b1971ec8eee044
Jun 19 06:33:54 dist-test-slave-nf4n krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:54.142158 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:54.122862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:58281 (local address 127.28.104.254:34261)
0619 06:33:54.123100 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:54.123107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:54.124393 (+ 1286us) server_negotiation.cc:407] Connection header received
0619 06:33:54.126495 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:54.126501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:54.126681 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:54.126803 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:54.128899 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:54.129920 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:54.131753 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:54.132195 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:54.132347 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:54.136124 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:54.136131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:54.136135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:54.136173 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:54.138901 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:54.139670 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:54.139676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:54.139679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:54.139814 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:54.140243 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:54.140250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:54.140255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:54.140533 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:54.140720 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:54.141567 (+ 847us) server_negotiation.cc:299] Negotiation successful
0619 06:33:54.141807 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 06:33:54.143558 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:54.123433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34261 (local address 127.28.104.195:58281)
0619 06:33:54.124205 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:54.124254 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:54.126170 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:54.127068 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:54.127081 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:54.127790 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:54.128700 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:54.128736 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:54.130091 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:54.130097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:54.131570 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:54.131599 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:54.132438 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:54.132448 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:54.132732 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:54.133519 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:54.133545 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:54.135841 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:54.139065 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:54.139075 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:54.139092 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:54.139534 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:54.139929 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:54.139934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:54.139938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:54.140129 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:54.140677 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:54.140685 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:54.141099 (+ 414us) client_negotiation.cc:769] Sending connection context
0619 06:33:54.141444 (+ 345us) client_negotiation.cc:240] Negotiation successful
0619 06:33:54.142068 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":303,"threads_started":1}
I20260619 06:33:54.144824 32003 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34261
I20260619 06:33:54.145169 32003 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:54.145861 32003 heartbeater.cc:511] Master 127.28.104.254:34261 requested a full tablet report, sending...
I20260619 06:33:54.147192 31547 ts_manager.cc:194] Registered new tserver with Master: 6ee290176374460a83665a5f2ddba649 (127.28.104.195:33213)
I20260619 06:33:54.147992 31547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:58281
I20260619 06:33:54.155126 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:33:54.163251 31736 heartbeater.cc:503] Master 127.28.104.254:34261 was elected leader, sending a full tablet report...
W20260619 06:33:54.187318 31738 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:54.184507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.28.104.254:34261)
0619 06:33:54.184749 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:54.184758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:54.185150 (+ 392us) server_negotiation.cc:407] Connection header received
0619 06:33:54.185444 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:54.185451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:54.185540 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:54.185636 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:54.187024 (+ 1388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34976: BlockingRecv error: recv got EOF from 127.0.0.1:34976 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:33:54.190398 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31608
W20260619 06:33:54.193275 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31608: Not authorized: Client connection negotiation failed: client connection to 127.28.104.193:35107: server requires authentication, but client does not have Kerberos credentials available
I20260619 06:33:54.193504 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 31608
I20260619 06:33:54.209533 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31742
W20260619 06:33:54.212312 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31742: Not authorized: Client connection negotiation failed: client connection to 127.28.104.194:36049: server requires authentication, but client does not have Kerberos credentials available
I20260619 06:33:54.212436 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 31742
I20260619 06:33:54.227427 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31875
W20260619 06:33:54.230496 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31875: Not authorized: Client connection negotiation failed: client connection to 127.28.104.195:33213: server requires authentication, but client does not have Kerberos credentials available
I20260619 06:33:54.230697 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 31875
I20260619 06:33:54.246261 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 31515
I20260619 06:33:54.247675 31577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:54.350862 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 31515
2026-06-19T06:33:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3526 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:33:54 dist-test-slave-nf4n krb5kdc[32020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:33:54 dist-test-slave-nf4n krb5kdc[32020](info): set up 2 sockets
Jun 19 06:33:54 dist-test-slave-nf4n krb5kdc[32020](info): commencing operation
krb5kdc: starting...
W20260619 06:33:55.440999 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:33:55 dist-test-slave-nf4n krb5kdc[32020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:33:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:33:55.649147 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:38447
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:38447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:55.975088 32033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:55.975620 32033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:55.975826 32033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:55.983472 32033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:55.983666 32033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:55.983795 32033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:55.983912 32033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:55.984017 32033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:55.995136 32033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:38447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:38447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:55.996691 32033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:55.998209 32033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:56.011210 32041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:56.011180 32039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:56.011426 32038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:56.012318 32033 server_base.cc:1034] running on GCE node
I20260619 06:33:56.013244 32033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:56.014753 32033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:56.015966 32033 hybrid_clock.cc:648] HybridClock initialized: now 1781850836015898 us; error 53 us; skew 500 ppm
I20260619 06:33:56.016396 32033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:56 dist-test-slave-nf4n krb5kdc[32020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850836, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:56.019927 32033 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:33:56.021580 32033 webserver.cc:460] Webserver started at http://127.28.104.254:41775/ using document root <none> and password file <none>
I20260619 06:33:56.022348 32033 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:56.022512 32033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:56.022856 32033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:56.026275 32033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "8b67961dff64461697c9a97feb1f9f13"
format_stamp: "Formatted at 2026-06-19 06:33:56 on dist-test-slave-nf4n"
server_key: "5ad9417f8d7fcb05cff39a69cc89e4b5"
server_key_iv: "d7015f2e36dd86f2481ac32ab79a6b62"
server_key_version: "encryptionkey@0"
I20260619 06:33:56.027175 32033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "8b67961dff64461697c9a97feb1f9f13"
format_stamp: "Formatted at 2026-06-19 06:33:56 on dist-test-slave-nf4n"
server_key: "5ad9417f8d7fcb05cff39a69cc89e4b5"
server_key_iv: "d7015f2e36dd86f2481ac32ab79a6b62"
server_key_version: "encryptionkey@0"
I20260619 06:33:56.032771 32033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:33:56.036427 32048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:56.037916 32033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:56.038107 32033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "8b67961dff64461697c9a97feb1f9f13"
format_stamp: "Formatted at 2026-06-19 06:33:56 on dist-test-slave-nf4n"
server_key: "5ad9417f8d7fcb05cff39a69cc89e4b5"
server_key_iv: "d7015f2e36dd86f2481ac32ab79a6b62"
server_key_version: "encryptionkey@0"
I20260619 06:33:56.038342 32033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:56.058331 32033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:56.062839 32033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:56.063146 32033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:56.081858 32033 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:38447
I20260619 06:33:56.081946 32099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:38447 every 8 connection(s)
I20260619 06:33:56.083267 32033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:33:56.087203 32100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:56.087334 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 32033
I20260619 06:33:56.087589 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:33:56.088317 29091 external_mini_cluster.cc:1428] Setting key 70f36b55a755e12fe5d9b043e6a3ce9f
I20260619 06:33:56.101506 32100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13: Bootstrap starting.
Jun 19 06:33:56 dist-test-slave-nf4n krb5kdc[32020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:56.105430 32100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:56.106909 32100 log.cc:824] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:56.110282 32100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13: No bootstrap required, opened a new log
I20260619 06:33:56.113233 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:56.090245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.28.104.254:38447)
0619 06:33:56.091076 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:56.091104 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:56.091144 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:33:56.093537 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:56.093560 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:56.093963 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:56.094304 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:56.096169 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:56.097536 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:56.099362 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:56.100010 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:56.100144 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:56.104720 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:56.104731 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:56.104751 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:56.104798 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:56.108635 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:56.109403 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:56.109412 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:56.109422 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:56.109624 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:56.109997 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:56.110002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:56.110005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:56.110705 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:56.111037 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:56.111357 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 06:33:56.111782 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":403,"threads_started":1}
I20260619 06:33:56.117138 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:33:56.119521 32100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b67961dff64461697c9a97feb1f9f13" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38447 } }
I20260619 06:33:56.119933 32100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:56.120041 32100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b67961dff64461697c9a97feb1f9f13, State: Initialized, Role: FOLLOWER
I20260619 06:33:56.120675 32100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b67961dff64461697c9a97feb1f9f13" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38447 } }
I20260619 06:33:56.120921 32100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:33:56.121088 32100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:33:56.121268 32100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:56.123474 32100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b67961dff64461697c9a97feb1f9f13" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38447 } }
I20260619 06:33:56.124023 32100 leader_election.cc:302] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b67961dff64461697c9a97feb1f9f13; no voters:
I20260619 06:33:56.124651 32100 leader_election.cc:288] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:33:56.124815 32106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:56.126802 32100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:56.129621 32106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [term 1 LEADER]: Becoming Leader. State: Replica: 8b67961dff64461697c9a97feb1f9f13, State: Running, Role: LEADER
I20260619 06:33:56.130347 32106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b67961dff64461697c9a97feb1f9f13" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38447 } }
I20260619 06:33:56.133867 32107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b67961dff64461697c9a97feb1f9f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b67961dff64461697c9a97feb1f9f13" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38447 } } }
I20260619 06:33:56.134254 32107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:56.134836 32108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b67961dff64461697c9a97feb1f9f13. Latest consensus state: current_term: 1 leader_uuid: "8b67961dff64461697c9a97feb1f9f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b67961dff64461697c9a97feb1f9f13" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38447 } } }
I20260619 06:33:56.135102 32108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13 [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:56.156368 32118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:33:56.170164 32118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:33:56.180027 32128 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:33:56.183195 32118 catalog_manager.cc:1347] Generated new cluster ID: 3877bc85ce7745ae957ff3ff25323071
I20260619 06:33:56.183314 32118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:33:56.189954 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:56.182212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.28.104.254:38447)
0619 06:33:56.182394 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:56.182403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:56.183060 (+ 657us) server_negotiation.cc:407] Connection header received
0619 06:33:56.183362 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:56.183368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:56.183448 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:56.183525 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:56.184656 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:56.185467 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:56.186755 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:56.187199 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:56.187335 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:56.188452 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:56.188457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:56.188461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:33:56.189121 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:56.189190 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:33:56.189214 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:56.189334 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:56.189471 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 06:33:56.189572 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:33:56.199484 32118 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:33:56.201402 32118 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:33:56.211169 32118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b67961dff64461697c9a97feb1f9f13: Generated new TSK 0
I20260619 06:33:56.212072 32118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:33:56.221810 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 32033
I20260619 06:33:56.223147 32095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:56.313395 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 32033
2026-06-19T06:33:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1959 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:33:56 dist-test-slave-nf4n krb5kdc[32135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:33:56 dist-test-slave-nf4n krb5kdc[32135](info): set up 2 sockets
Jun 19 06:33:56 dist-test-slave-nf4n krb5kdc[32135](info): commencing operation
krb5kdc: starting...
W20260619 06:33:57.513373 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:33:57 dist-test-slave-nf4n krb5kdc[32135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:33:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:33:57.705562 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:44049
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:44049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:58.028838 32148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:58.029232 32148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:58.029426 32148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:58.036746 32148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:58.036851 32148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:58.036969 32148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:58.037101 32148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:58.037161 32148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:58.048379 32148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:44049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:44049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:58.049861 32148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:58.051259 32148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:58.063004 32154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:58.063050 32156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:58.063686 32153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:58.064975 32148 server_base.cc:1034] running on GCE node
I20260619 06:33:58.065785 32148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:58.067163 32148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:58.068362 32148 hybrid_clock.cc:648] HybridClock initialized: now 1781850838068317 us; error 32 us; skew 500 ppm
I20260619 06:33:58.068706 32148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:58 dist-test-slave-nf4n krb5kdc[32135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850838, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:58.071911 32148 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:33:58.073390 32148 webserver.cc:460] Webserver started at http://127.28.104.254:37277/ using document root <none> and password file <none>
I20260619 06:33:58.074034 32148 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:58.074137 32148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:58.074421 32148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:58.077407 32148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "33df04a231724b9eab1ded3161ff139d"
format_stamp: "Formatted at 2026-06-19 06:33:58 on dist-test-slave-nf4n"
server_key: "7b04c456f219ad0a1f0c0764d8d330ec"
server_key_iv: "a0a441be666261f02c62715ce15874bb"
server_key_version: "encryptionkey@0"
I20260619 06:33:58.078104 32148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "33df04a231724b9eab1ded3161ff139d"
format_stamp: "Formatted at 2026-06-19 06:33:58 on dist-test-slave-nf4n"
server_key: "7b04c456f219ad0a1f0c0764d8d330ec"
server_key_iv: "a0a441be666261f02c62715ce15874bb"
server_key_version: "encryptionkey@0"
I20260619 06:33:58.083081 32148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:33:58.086426 32163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:58.087751 32148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:33:58.087924 32148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "33df04a231724b9eab1ded3161ff139d"
format_stamp: "Formatted at 2026-06-19 06:33:58 on dist-test-slave-nf4n"
server_key: "7b04c456f219ad0a1f0c0764d8d330ec"
server_key_iv: "a0a441be666261f02c62715ce15874bb"
server_key_version: "encryptionkey@0"
I20260619 06:33:58.088119 32148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:58.102761 32148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:58.107288 32148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:58.107578 32148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:58.125874 32148 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:44049
I20260619 06:33:58.125903 32214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:44049 every 8 connection(s)
I20260619 06:33:58.127262 32148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:33:58.130966 32215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:58.134359 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 32148
I20260619 06:33:58.134591 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:33:58.135383 29091 external_mini_cluster.cc:1428] Setting key 512eee7cd833872035262d4ef2f91ac6
I20260619 06:33:58.144047 32215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d: Bootstrap starting.
I20260619 06:33:58.147946 32215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:58.149343 32215 log.cc:824] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:33:58 dist-test-slave-nf4n krb5kdc[32135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:58.152596 32215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d: No bootstrap required, opened a new log
I20260619 06:33:58.158713 32218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:58.137478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.28.104.254:44049)
0619 06:33:58.138236 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:58.138253 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:58.138294 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:33:58.139911 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:58.139934 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:58.140359 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:58.140688 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:58.142529 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:58.143827 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:58.145541 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:58.146211 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:58.146356 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:58.151336 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:58.151343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:58.151357 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:58.151391 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:58.154648 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:58.155440 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:58.155446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:58.155452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:58.155616 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:58.155998 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:58.156001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:58.156003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:58.156516 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:58.156752 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:58.157228 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 06:33:58.157677 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":357,"threads_started":1}
I20260619 06:33:58.161856 32215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33df04a231724b9eab1ded3161ff139d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44049 } }
I20260619 06:33:58.162240 32215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:58.162359 32215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33df04a231724b9eab1ded3161ff139d, State: Initialized, Role: FOLLOWER
I20260619 06:33:58.162901 32215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33df04a231724b9eab1ded3161ff139d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44049 } }
I20260619 06:33:58.163137 32215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:33:58.163282 32215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:33:58.163471 32215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:58.165755 32215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33df04a231724b9eab1ded3161ff139d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44049 } }
I20260619 06:33:58.166237 32215 leader_election.cc:302] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33df04a231724b9eab1ded3161ff139d; no voters:
I20260619 06:33:58.166826 32215 leader_election.cc:288] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:33:58.166951 32220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:58.167618 32220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [term 1 LEADER]: Becoming Leader. State: Replica: 33df04a231724b9eab1ded3161ff139d, State: Running, Role: LEADER
I20260619 06:33:58.168218 32220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33df04a231724b9eab1ded3161ff139d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44049 } }
I20260619 06:33:58.168967 32215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:58.170810 32222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33df04a231724b9eab1ded3161ff139d. Latest consensus state: current_term: 1 leader_uuid: "33df04a231724b9eab1ded3161ff139d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33df04a231724b9eab1ded3161ff139d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44049 } } }
I20260619 06:33:58.171355 32222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:58.171236 32221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33df04a231724b9eab1ded3161ff139d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33df04a231724b9eab1ded3161ff139d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44049 } } }
I20260619 06:33:58.172610 32229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:33:58.173163 32221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:58.179735 32229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:33:58.188853 32229 catalog_manager.cc:1347] Generated new cluster ID: acd093c67c314feab795b442ba6f64c9
I20260619 06:33:58.189041 32229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:33:58.206818 32229 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:33:58.208389 32229 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:33:58.219305 32229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33df04a231724b9eab1ded3161ff139d: Generated new TSK 0
I20260619 06:33:58.220189 32229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:33:58.272089 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44049
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:58.598784 32241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:58.599071 32241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:58.599164 32241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:58.606390 32241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:58.606513 32241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:58.606628 32241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:33:58.616539 32241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:58.617801 32241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:58.619138 32241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:58.632457 32246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:58.632400 32249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:58.632375 32247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:58.633291 32241 server_base.cc:1034] running on GCE node
I20260619 06:33:58.633898 32241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:58.634840 32241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:58.636065 32241 hybrid_clock.cc:648] HybridClock initialized: now 1781850838635996 us; error 64 us; skew 500 ppm
I20260619 06:33:58.636375 32241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:58 dist-test-slave-nf4n krb5kdc[32135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850838, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:58.643461 32241 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:33:58.645015 32241 webserver.cc:460] Webserver started at http://127.28.104.193:42495/ using document root <none> and password file <none>
I20260619 06:33:58.645777 32241 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:58.645922 32241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:58.646229 32241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:58.649168 32241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "6cda4b6d77c143cf99a7adf543538962"
format_stamp: "Formatted at 2026-06-19 06:33:58 on dist-test-slave-nf4n"
server_key: "1639bd68f649ac8631b40569aaccbb7b"
server_key_iv: "40930b2315bd406a7252df258eab193a"
server_key_version: "encryptionkey@0"
I20260619 06:33:58.649938 32241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "6cda4b6d77c143cf99a7adf543538962"
format_stamp: "Formatted at 2026-06-19 06:33:58 on dist-test-slave-nf4n"
server_key: "1639bd68f649ac8631b40569aaccbb7b"
server_key_iv: "40930b2315bd406a7252df258eab193a"
server_key_version: "encryptionkey@0"
I20260619 06:33:58.655045 32241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:33:58.658478 32256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:58.659904 32241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:33:58.660039 32241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "6cda4b6d77c143cf99a7adf543538962"
format_stamp: "Formatted at 2026-06-19 06:33:58 on dist-test-slave-nf4n"
server_key: "1639bd68f649ac8631b40569aaccbb7b"
server_key_iv: "40930b2315bd406a7252df258eab193a"
server_key_version: "encryptionkey@0"
I20260619 06:33:58.660202 32241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:58.677014 32241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:58.681257 32241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:58.681504 32241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:58.682426 32241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:58.683698 32241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:58.683779 32241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:58.683863 32241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:58.683909 32241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:58.714730 32241 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:45837
I20260619 06:33:58.714768 32368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:45837 every 8 connection(s)
I20260619 06:33:58.716029 32241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:33:58.720916 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 32241
I20260619 06:33:58.721177 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:33:58.721895 29091 external_mini_cluster.cc:1428] Setting key 3c139742dc6386ac1b9e2f4380e69151
Jun 19 06:33:58 dist-test-slave-nf4n krb5kdc[32135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850838, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:58.739575 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:58.718715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:50085 (local address 127.28.104.254:44049)
0619 06:33:58.719211 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:58.719216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:58.720276 (+ 1060us) server_negotiation.cc:407] Connection header received
0619 06:33:58.722241 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:58.722247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:58.722338 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:58.722474 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:58.724400 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:58.725217 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:58.727739 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:58.728174 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:58.728274 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:58.731921 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:58.731932 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:58.731950 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:58.732007 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:58.735750 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:58.736498 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:58.736508 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:58.736515 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:58.736688 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:58.737209 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:58.737217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:58.737223 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:58.737655 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:58.737833 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:58.738705 (+ 872us) server_negotiation.cc:299] Negotiation successful
0619 06:33:58.739003 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
I20260619 06:33:58.740072 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:58.719366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:44049 (local address 127.28.104.193:50085)
0619 06:33:58.720102 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:58.720148 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:58.721960 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:58.722774 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:58.722786 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:58.723370 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:58.724206 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:58.724234 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:58.725383 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:58.725392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:58.727576 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:58.727594 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:58.728403 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:58.728412 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:58.728740 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:58.729519 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:58.729541 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:58.731684 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:58.735909 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:58.735916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:58.735931 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:58.736365 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:58.736852 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:58.736856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:58.736858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:58.737091 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:58.737795 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:58.737802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:58.738279 (+ 477us) client_negotiation.cc:769] Sending connection context
0619 06:33:58.738511 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 06:33:58.738920 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":273,"threads_started":1}
I20260619 06:33:58.741849 32369 heartbeater.cc:347] Connected to a master server at 127.28.104.254:44049
I20260619 06:33:58.742293 32369 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
I20260619 06:33:58.743312 32369 heartbeater.cc:511] Master 127.28.104.254:44049 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
I20260619 06:33:58.746088 32180 ts_manager.cc:194] Registered new tserver with Master: 6cda4b6d77c143cf99a7adf543538962 (127.28.104.193:45837)
I20260619 06:33:58.748454 32180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:50085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:33:58.770639 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44049
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:59.106982 32375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:59.107267 32375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:59.107455 32375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:59.114889 32375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:59.115000 32375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:59.115213 32375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:33:59.124980 32375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:59.126382 32375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:59.127679 32375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:59.141784 32380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:59.142122 32375 server_base.cc:1034] running on GCE node
W20260619 06:33:59.141748 32383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:59.141713 32381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:59.143247 32375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:59.144255 32375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:59.145483 32375 hybrid_clock.cc:648] HybridClock initialized: now 1781850839145414 us; error 58 us; skew 500 ppm
I20260619 06:33:59.145803 32375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:59 dist-test-slave-nf4n krb5kdc[32135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:59.148937 32375 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:33:59.150445 32375 webserver.cc:460] Webserver started at http://127.28.104.194:38751/ using document root <none> and password file <none>
I20260619 06:33:59.151100 32375 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:59.151232 32375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:59.151564 32375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:59.154752 32375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "992cf061f4204385aeb6f33fe9f1e9de"
format_stamp: "Formatted at 2026-06-19 06:33:59 on dist-test-slave-nf4n"
server_key: "6bc27c2fa25b66f3e7cf11e2e3a19c04"
server_key_iv: "b469915d8453d64a72729a220409946b"
server_key_version: "encryptionkey@0"
I20260619 06:33:59.155525 32375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "992cf061f4204385aeb6f33fe9f1e9de"
format_stamp: "Formatted at 2026-06-19 06:33:59 on dist-test-slave-nf4n"
server_key: "6bc27c2fa25b66f3e7cf11e2e3a19c04"
server_key_iv: "b469915d8453d64a72729a220409946b"
server_key_version: "encryptionkey@0"
I20260619 06:33:59.160866 32375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:33:59.164573 32390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:59.166008 32375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:59.166177 32375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "992cf061f4204385aeb6f33fe9f1e9de"
format_stamp: "Formatted at 2026-06-19 06:33:59 on dist-test-slave-nf4n"
server_key: "6bc27c2fa25b66f3e7cf11e2e3a19c04"
server_key_iv: "b469915d8453d64a72729a220409946b"
server_key_version: "encryptionkey@0"
I20260619 06:33:59.166375 32375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:59.181529 32375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:59.185707 32375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:59.185959 32375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:59.186852 32375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:59.188174 32375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:59.188243 32375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:59.188333 32375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:59.188377 32375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:59.221623 32375 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:39831
I20260619 06:33:59.221715 32502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:39831 every 8 connection(s)
I20260619 06:33:59.222985 32375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:33:59.224134 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 32375
I20260619 06:33:59.224355 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:33:59.224973 29091 external_mini_cluster.cc:1428] Setting key 41e8560588714cd9cde53bc8c98bb62e
Jun 19 06:33:59 dist-test-slave-nf4n krb5kdc[32135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
I20260619 06:33:59.247663 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:59.226001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:43465 (local address 127.28.104.254:44049)
0619 06:33:59.226206 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:59.226210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:59.227805 (+ 1595us) server_negotiation.cc:407] Connection header received
0619 06:33:59.230798 (+ 2993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:59.230803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:59.230875 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:59.230955 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:59.233244 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.234218 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.235768 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.236553 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.236694 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:59.240225 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:59.240236 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:59.240244 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:59.240292 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:59.244308 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.245189 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.245195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.245199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.245376 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.245869 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.245875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.245878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.246262 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:59.246381 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:59.247197 (+ 816us) server_negotiation.cc:299] Negotiation successful
0619 06:33:59.247406 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
I20260619 06:33:59.248951 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:59.226684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:44049 (local address 127.28.104.194:43465)
0619 06:33:59.227604 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:59.227673 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:59.230453 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:59.231269 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:59.231284 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:59.232134 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:59.233049 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:59.233080 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.234379 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.234383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:59.235615 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:59.235627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.236787 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.236792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:59.236999 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:59.237786 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:59.237809 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:59.239985 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:59.244495 (+ 4510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:59.244503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:59.244519 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:59.244983 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:59.245492 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:59.245495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:59.245498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:59.245721 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:59.246404 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:59.246411 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:59.246777 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 06:33:59.247087 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 06:33:59.247524 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":358,"threads_started":1}
I20260619 06:33:59.250717 32503 heartbeater.cc:347] Connected to a master server at 127.28.104.254:44049
I20260619 06:33:59.251061 32503 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:59.252015 32503 heartbeater.cc:511] Master 127.28.104.254:44049 requested a full tablet report, sending...
I20260619 06:33:59.253746 32180 ts_manager.cc:194] Registered new tserver with Master: 992cf061f4204385aeb6f33fe9f1e9de (127.28.104.194:39831)
I20260619 06:33:59.254771 32180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:43465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:33:59.274176 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44049
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:59.596627 32508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:59.596920 32508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:59.597126 32508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:59.604332 32508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:59.604442 32508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:59.604624 32508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:33:59.614292 32508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:59.615739 32508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:59.617081 32508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:59.630563 32516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:59.630600 32514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:59.630522 32513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:59.631521 32508 server_base.cc:1034] running on GCE node
I20260619 06:33:59.632172 32508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:59.633189 32508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:59.634413 32508 hybrid_clock.cc:648] HybridClock initialized: now 1781850839634352 us; error 57 us; skew 500 ppm
I20260619 06:33:59.634723 32508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:59 dist-test-slave-nf4n krb5kdc[32135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:59.637797 32508 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:33:59.639333 32508 webserver.cc:460] Webserver started at http://127.28.104.195:45421/ using document root <none> and password file <none>
I20260619 06:33:59.640014 32508 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:59.640156 32508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:59.640491 32508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:59.643432 32508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "e68ce889c36a40d4a3e81c52b569ab43"
format_stamp: "Formatted at 2026-06-19 06:33:59 on dist-test-slave-nf4n"
server_key: "6c9526e3123b87a977810b24e50871c0"
server_key_iv: "0eb5d2e7a2824758bf2c952a377622c8"
server_key_version: "encryptionkey@0"
I20260619 06:33:59.644188 32508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "e68ce889c36a40d4a3e81c52b569ab43"
format_stamp: "Formatted at 2026-06-19 06:33:59 on dist-test-slave-nf4n"
server_key: "6c9526e3123b87a977810b24e50871c0"
server_key_iv: "0eb5d2e7a2824758bf2c952a377622c8"
server_key_version: "encryptionkey@0"
I20260619 06:33:59.649202 32508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 06:33:59.652762 32523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:59.654112 32508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:59.654291 32508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "e68ce889c36a40d4a3e81c52b569ab43"
format_stamp: "Formatted at 2026-06-19 06:33:59 on dist-test-slave-nf4n"
server_key: "6c9526e3123b87a977810b24e50871c0"
server_key_iv: "0eb5d2e7a2824758bf2c952a377622c8"
server_key_version: "encryptionkey@0"
I20260619 06:33:59.654495 32508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:59.667290 32508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:59.671564 32508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:59.671875 32508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:59.672894 32508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:33:59.674278 32508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:33:59.674350 32508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:59.674430 32508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:33:59.674474 32508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:59.708099 32508 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:36007
I20260619 06:33:59.708135 32635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:36007 every 8 connection(s)
I20260619 06:33:59.709520 32508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:33:59.712745 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 32508
I20260619 06:33:59.712978 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:33:59.713608 29091 external_mini_cluster.cc:1428] Setting key 46bf0cc93811ad835dab210ecf225bea
Jun 19 06:33:59 dist-test-slave-nf4n krb5kdc[32135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:59.730551 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:59.712176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:52815 (local address 127.28.104.254:44049)
0619 06:33:59.712425 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:59.712431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:59.713873 (+ 1442us) server_negotiation.cc:407] Connection header received
0619 06:33:59.716218 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:59.716225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:59.716314 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:59.716422 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:59.718131 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.719220 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.720706 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.721175 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.721307 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:59.724808 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:59.724814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:59.724818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:59.724853 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:59.727410 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.728304 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.728312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.728316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.728456 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.728879 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.728882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.728884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.729312 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:59.729430 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:59.730134 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 06:33:59.730308 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 06:33:59.731555 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:59.712739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:44049 (local address 127.28.104.195:52815)
0619 06:33:59.713690 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:59.713742 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:59.715902 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:59.716685 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:59.716698 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:59.717380 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:59.717945 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:59.717961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.719377 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.719381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:59.720579 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:59.720590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.721423 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.721429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:59.721638 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:59.722354 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:59.722377 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:59.724578 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:59.727571 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:59.727581 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:59.727599 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:59.728186 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:59.728571 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:59.728577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:59.728580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:59.728771 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:59.729468 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:59.729478 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:59.729819 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 06:33:59.730064 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 06:33:59.730462 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":376,"threads_started":1}
I20260619 06:33:59.732990 32636 heartbeater.cc:347] Connected to a master server at 127.28.104.254:44049
I20260619 06:33:59.733392 32636 heartbeater.cc:464] Registering TS with master...
I20260619 06:33:59.734143 32636 heartbeater.cc:511] Master 127.28.104.254:44049 requested a full tablet report, sending...
I20260619 06:33:59.735569 32180 ts_manager.cc:194] Registered new tserver with Master: e68ce889c36a40d4a3e81c52b569ab43 (127.28.104.195:36007)
I20260619 06:33:59.736387 32180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:52815
I20260619 06:33:59.741711 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:33:59.751952 32369 heartbeater.cc:503] Master 127.28.104.254:44049 was elected leader, sending a full tablet report...
Jun 19 06:33:59 dist-test-slave-nf4n krb5kdc[32135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850839, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 06:33:59 dist-test-slave-nf4n krb5kdc[32135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850839, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:33:59.790236 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:59.776288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.28.104.254:44049)
0619 06:33:59.776497 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:59.776506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:59.776924 (+ 418us) server_negotiation.cc:407] Connection header received
0619 06:33:59.777190 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:59.777195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:59.777268 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:59.777374 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:59.778715 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.779459 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.780727 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.781231 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.781378 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:59.785244 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:59.785256 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:59.785260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:59.785294 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:59.787619 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.788262 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.788267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.788272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.788409 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.788789 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.788796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.788800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.789170 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:59.789431 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:59.789706 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 06:33:59.789923 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 06:33:59.790782 32180 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45928
I20260619 06:33:59.792460 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 32241
Jun 19 06:33:59 dist-test-slave-nf4n krb5kdc[32135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850839, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:33:59.807260 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:59.793613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.28.104.193:45837)
0619 06:33:59.794231 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:59.794239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:59.794270 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:33:59.794328 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:59.794334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:59.794544 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:59.794663 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:59.795695 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.796664 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.797849 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.798426 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.798520 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:59.802322 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:59.802333 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:59.802348 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:59.802378 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:59.804946 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.805574 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.805580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.805582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.805694 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.806007 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.806010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.806012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.806334 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:59.806543 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:59.806776 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 06:33:59.806967 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":312,"threads_started":1}
W20260619 06:33:59.808521 32364 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50040
W20260619 06:33:59.809729 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 32241: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:33:59.809880 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 32241
I20260619 06:33:59.825521 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 32375
Jun 19 06:33:59 dist-test-slave-nf4n krb5kdc[32135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850839, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:33:59.841533 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:59.826482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.28.104.194:39831)
0619 06:33:59.827107 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:59.827114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:59.827142 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:33:59.827213 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:59.827219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:59.827439 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:59.827574 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:59.828718 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.829789 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.831055 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.831636 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.831746 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:59.835860 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:59.835872 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:59.835887 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:59.835923 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:59.838969 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.839582 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.839590 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.839592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.839722 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.840014 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.840017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.840019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.840355 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:59.840575 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:59.840814 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 06:33:59.841086 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":339,"threads_started":1}
W20260619 06:33:59.842869 32498 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56700
W20260619 06:33:59.843991 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 32375: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:33:59.844172 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 32375
I20260619 06:33:59.859840 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 32508
Jun 19 06:33:59 dist-test-slave-nf4n krb5kdc[32135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850839, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:33:59.876705 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:59.861109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.28.104.195:36007)
0619 06:33:59.861811 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:59.861818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:59.861848 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:33:59.861915 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:59.861922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:59.862165 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:59.862296 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:59.863580 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.864652 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.865893 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:59.866484 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:59.866589 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:59.871108 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:59.871120 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:59.871136 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:59.871169 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:59.873962 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.874606 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.874612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.874614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.874726 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:59.875094 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:59.875097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:59.875099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:59.875425 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:59.875655 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:59.876205 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 06:33:59.876422 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":370,"threads_started":1}
W20260619 06:33:59.878103 32631 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53412
W20260619 06:33:59.879238 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 32508: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:33:59.879402 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 32508
I20260619 06:33:59.895192 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 32148
I20260619 06:33:59.896345 32210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:00.003659 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 32148
2026-06-19T06:34:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3694 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:34:00 dist-test-slave-nf4n krb5kdc[32653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:34:00 dist-test-slave-nf4n krb5kdc[32653](info): set up 2 sockets
Jun 19 06:34:00 dist-test-slave-nf4n krb5kdc[32653](info): commencing operation
krb5kdc: starting...
W20260619 06:34:01.097301 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:34:01 dist-test-slave-nf4n krb5kdc[32653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:34:01.298496 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:36817
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:36817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:01.623914 32666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:01.624220 32666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:01.624313 32666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:01.631656 32666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:01.631780 32666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:01.631829 32666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:01.631886 32666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:01.631930 32666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:01.641949 32666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:36817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:01.643280 32666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:01.644618 32666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:01.656986 32672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:01.657087 32671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:01.657112 32674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:01.657950 32666 server_base.cc:1034] running on GCE node
I20260619 06:34:01.658789 32666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:01.660418 32666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:01.661669 32666 hybrid_clock.cc:648] HybridClock initialized: now 1781850841661607 us; error 56 us; skew 500 ppm
I20260619 06:34:01.662024 32666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:01 dist-test-slave-nf4n krb5kdc[32653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:01.665311 32666 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:34:01.666822 32666 webserver.cc:460] Webserver started at http://127.28.104.254:38103/ using document root <none> and password file <none>
I20260619 06:34:01.667500 32666 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:01.667596 32666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:01.667881 32666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:01.670893 32666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "350fe6845485419aa3a99f980ebf4f75"
format_stamp: "Formatted at 2026-06-19 06:34:01 on dist-test-slave-nf4n"
server_key: "927e891378a66a0337269fd6260b2b3f"
server_key_iv: "51c55d31809e73e6f4eee9bbff8d51d2"
server_key_version: "encryptionkey@0"
I20260619 06:34:01.671648 32666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "350fe6845485419aa3a99f980ebf4f75"
format_stamp: "Formatted at 2026-06-19 06:34:01 on dist-test-slave-nf4n"
server_key: "927e891378a66a0337269fd6260b2b3f"
server_key_iv: "51c55d31809e73e6f4eee9bbff8d51d2"
server_key_version: "encryptionkey@0"
I20260619 06:34:01.677088 32666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 06:34:01.680634 32681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:01.682523 32666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:34:01.682714 32666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "350fe6845485419aa3a99f980ebf4f75"
format_stamp: "Formatted at 2026-06-19 06:34:01 on dist-test-slave-nf4n"
server_key: "927e891378a66a0337269fd6260b2b3f"
server_key_iv: "51c55d31809e73e6f4eee9bbff8d51d2"
server_key_version: "encryptionkey@0"
I20260619 06:34:01.682934 32666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:01.694638 32666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:01.698799 32666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:01.699087 32666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:01.718438 32666 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:36817
I20260619 06:34:01.718456 32732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:36817 every 8 connection(s)
I20260619 06:34:01.719942 32666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:01.723846 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:01.727850 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 32666
I20260619 06:34:01.728060 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:01.728883 29091 external_mini_cluster.cc:1428] Setting key b854a339528c40291d0cb5fc0c210115
I20260619 06:34:01.737953 32733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75: Bootstrap starting.
I20260619 06:34:01.741999 32733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:01.743409 32733 log.cc:824] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:34:01 dist-test-slave-nf4n krb5kdc[32653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:01.747253 32733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75: No bootstrap required, opened a new log
I20260619 06:34:01.753135 32736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:01.731080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.28.104.254:36817)
0619 06:34:01.731883 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:01.731895 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:01.731938 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:34:01.733599 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:01.733624 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:01.734061 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:01.734407 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:01.736309 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:01.737816 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:01.739568 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:01.740305 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:01.740442 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:01.745394 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:01.745403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:01.745419 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:01.745460 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:01.748644 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:01.749468 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:01.749475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:01.749482 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:01.749659 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:01.750041 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:01.750047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:01.750050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:01.750667 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:01.750979 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:01.751446 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 06:34:01.751904 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":390,"threads_started":1}
I20260619 06:34:01.756153 32733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350fe6845485419aa3a99f980ebf4f75" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36817 } }
I20260619 06:34:01.756639 32733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:01.756778 32733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350fe6845485419aa3a99f980ebf4f75, State: Initialized, Role: FOLLOWER
I20260619 06:34:01.757491 32733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350fe6845485419aa3a99f980ebf4f75" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36817 } }
I20260619 06:34:01.757771 32733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:01.757920 32733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:01.758107 32733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:01.761082 32733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350fe6845485419aa3a99f980ebf4f75" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36817 } }
I20260619 06:34:01.761708 32733 leader_election.cc:302] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 350fe6845485419aa3a99f980ebf4f75; no voters:
I20260619 06:34:01.762400 32733 leader_election.cc:288] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:01.762573 32738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:01.763321 32738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [term 1 LEADER]: Becoming Leader. State: Replica: 350fe6845485419aa3a99f980ebf4f75, State: Running, Role: LEADER
I20260619 06:34:01.764004 32738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350fe6845485419aa3a99f980ebf4f75" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36817 } }
I20260619 06:34:01.764884 32733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:01.767216 32740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "350fe6845485419aa3a99f980ebf4f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350fe6845485419aa3a99f980ebf4f75" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36817 } } }
I20260619 06:34:01.767688 32740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:01.767318 32739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 350fe6845485419aa3a99f980ebf4f75. Latest consensus state: current_term: 1 leader_uuid: "350fe6845485419aa3a99f980ebf4f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350fe6845485419aa3a99f980ebf4f75" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36817 } } }
I20260619 06:34:01.769421 32739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:01.770942 32744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:01.777429 32744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:01.788782 32744 catalog_manager.cc:1347] Generated new cluster ID: 4930d7a746364814bad919290e0d1b71
I20260619 06:34:01.788929 32744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:01.818940 32744 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:01.820937 32744 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:01.832473 32744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 350fe6845485419aa3a99f980ebf4f75: Generated new TSK 0
I20260619 06:34:01.833851 32744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:34:01.890709 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36817
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:02.232961 32759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:02.233371 32759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:02.233541 32759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:02.241113 32759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:02.241279 32759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:02.241456 32759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:34:02.251447 32759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:02.252974 32759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:02.254509 32759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:02.268014 32764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:02.268010 32765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:02.268148 32767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:02.268978 32759 server_base.cc:1034] running on GCE node
I20260619 06:34:02.269618 32759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:02.270608 32759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:02.271804 32759 hybrid_clock.cc:648] HybridClock initialized: now 1781850842271754 us; error 39 us; skew 500 ppm
I20260619 06:34:02.272126 32759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:02 dist-test-slave-nf4n krb5kdc[32653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850842, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:02.275362 32759 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:34:02.276870 32759 webserver.cc:460] Webserver started at http://127.28.104.193:42827/ using document root <none> and password file <none>
I20260619 06:34:02.277639 32759 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:02.277783 32759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:02.278092 32759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:02.280952 32759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "f657851b16304134bdfd0baabbd28bea"
format_stamp: "Formatted at 2026-06-19 06:34:02 on dist-test-slave-nf4n"
server_key: "82e5635f7d770440b3d796b8851b241c"
server_key_iv: "05f667d472fbc9ef79415ada68c8e25a"
server_key_version: "encryptionkey@0"
I20260619 06:34:02.281875 32759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "f657851b16304134bdfd0baabbd28bea"
format_stamp: "Formatted at 2026-06-19 06:34:02 on dist-test-slave-nf4n"
server_key: "82e5635f7d770440b3d796b8851b241c"
server_key_iv: "05f667d472fbc9ef79415ada68c8e25a"
server_key_version: "encryptionkey@0"
I20260619 06:34:02.287326 32759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:34:02.291059 306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:02.292493 32759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:34:02.292671 32759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "f657851b16304134bdfd0baabbd28bea"
format_stamp: "Formatted at 2026-06-19 06:34:02 on dist-test-slave-nf4n"
server_key: "82e5635f7d770440b3d796b8851b241c"
server_key_iv: "05f667d472fbc9ef79415ada68c8e25a"
server_key_version: "encryptionkey@0"
I20260619 06:34:02.292912 32759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:02.312752 32759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:02.317389 32759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:02.317699 32759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:02.318686 32759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:02.320060 32759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:02.320161 32759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:02.320336 32759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:02.320413 32759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:02.355294 32759 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:37053
I20260619 06:34:02.355347 419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:37053 every 8 connection(s)
I20260619 06:34:02.356601 32759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:34:02.362886 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 32759
I20260619 06:34:02.363097 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:34:02.363734 29091 external_mini_cluster.cc:1428] Setting key a8cf4975575d2e6a99fdbc92af310e36
Jun 19 06:34:02 dist-test-slave-nf4n krb5kdc[32653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850842, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:02.378497 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:02.359258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:46879 (local address 127.28.104.254:36817)
0619 06:34:02.359673 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:02.359677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:02.360682 (+ 1005us) server_negotiation.cc:407] Connection header received
0619 06:34:02.362748 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:02.362754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:02.362846 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:02.362968 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:02.364849 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:02.365863 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:02.367578 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:02.368325 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:02.368472 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:02.372520 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:02.372526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:02.372530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:02.372565 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:02.375264 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:02.376070 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:02.376076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:02.376082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:02.376206 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:02.376653 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:02.376659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:02.376662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:02.376944 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:02.377137 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:02.377869 (+ 732us) server_negotiation.cc:299] Negotiation successful
0619 06:34:02.378109 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":199,"threads_started":1}
I20260619 06:34:02.379616 423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:02.359811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:36817 (local address 127.28.104.193:46879)
0619 06:34:02.360511 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:02.360549 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:02.362450 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:02.363266 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:02.363275 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:02.363804 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:02.364677 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:02.364699 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:02.366037 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:02.366041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:02.367421 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:02.367434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:02.368549 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:02.368554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:02.368774 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:02.369632 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:02.369664 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:02.372275 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:02.375435 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:02.375446 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:02.375465 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:02.375939 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:02.376329 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:02.376335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:02.376338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:02.376538 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:02.377094 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:02.377102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:02.377489 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 06:34:02.377780 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 06:34:02.378359 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":277,"threads_started":1}
I20260619 06:34:02.381431 420 heartbeater.cc:347] Connected to a master server at 127.28.104.254:36817
I20260619 06:34:02.381744 420 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:02.382493 420 heartbeater.cc:511] Master 127.28.104.254:36817 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
I20260619 06:34:02.384865 32698 ts_manager.cc:194] Registered new tserver with Master: f657851b16304134bdfd0baabbd28bea (127.28.104.193:37053)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
I20260619 06:34:02.386942 32698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:46879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:34:02.410238 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36817
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:02.743341 426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:02.743623 426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:02.743781 426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:02.751201 426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:02.751312 426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:02.751505 426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:34:02.761484 426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:02.762956 426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:02.764340 426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:02.780051 431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:02.780299 434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:02.781174 426 server_base.cc:1034] running on GCE node
W20260619 06:34:02.781255 432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:02.781939 426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:02.783031 426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:02.784288 426 hybrid_clock.cc:648] HybridClock initialized: now 1781850842784241 us; error 36 us; skew 500 ppm
I20260619 06:34:02.784665 426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:02 dist-test-slave-nf4n krb5kdc[32653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850842, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:02.788256 426 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:34:02.789713 426 webserver.cc:460] Webserver started at http://127.28.104.194:39101/ using document root <none> and password file <none>
I20260619 06:34:02.790375 426 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:02.790469 426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:02.790723 426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:02.793635 426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "8039d4f121694c088bf65b58ab7299c5"
format_stamp: "Formatted at 2026-06-19 06:34:02 on dist-test-slave-nf4n"
server_key: "5a441e18fedd4b28e7f909cbe77aa464"
server_key_iv: "92a64dd40decb4dfc96d57ecfa21b13e"
server_key_version: "encryptionkey@0"
I20260619 06:34:02.794348 426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "8039d4f121694c088bf65b58ab7299c5"
format_stamp: "Formatted at 2026-06-19 06:34:02 on dist-test-slave-nf4n"
server_key: "5a441e18fedd4b28e7f909cbe77aa464"
server_key_iv: "92a64dd40decb4dfc96d57ecfa21b13e"
server_key_version: "encryptionkey@0"
I20260619 06:34:02.799683 426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:34:02.803179 441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:02.804585 426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:34:02.804769 426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "8039d4f121694c088bf65b58ab7299c5"
format_stamp: "Formatted at 2026-06-19 06:34:02 on dist-test-slave-nf4n"
server_key: "5a441e18fedd4b28e7f909cbe77aa464"
server_key_iv: "92a64dd40decb4dfc96d57ecfa21b13e"
server_key_version: "encryptionkey@0"
I20260619 06:34:02.805004 426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:02.827310 426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:02.831939 426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:02.832234 426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:02.833340 426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:02.834708 426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:02.834786 426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:02.834930 426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:02.834980 426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:02.868281 426 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:43719
I20260619 06:34:02.868368 554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:43719 every 8 connection(s)
I20260619 06:34:02.869685 426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:34:02.879629 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 426
I20260619 06:34:02.879827 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:34:02.880302 29091 external_mini_cluster.cc:1428] Setting key 706e3432d4f76102cdd323e1cd508e4e
Jun 19 06:34:02 dist-test-slave-nf4n krb5kdc[32653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850842, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:02.891151 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:02.872408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:59843 (local address 127.28.104.254:36817)
0619 06:34:02.872578 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:02.872583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:02.873932 (+ 1349us) server_negotiation.cc:407] Connection header received
0619 06:34:02.875744 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:02.875750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:02.875848 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:02.875970 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:02.877508 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:02.878267 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:02.879793 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:02.880286 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:02.880421 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:02.884174 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:02.884183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:02.884191 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:02.884237 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:02.887980 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:02.888756 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:02.888766 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:02.888769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:02.888893 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:02.889414 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:02.889419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:02.889422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:02.889704 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:02.889862 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:02.890559 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 06:34:02.890790 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:34:02.892167 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:02.872926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:36817 (local address 127.28.104.194:59843)
0619 06:34:02.873745 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:02.873790 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:02.875474 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:02.876198 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:02.876206 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:02.876699 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:02.877337 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:02.877359 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:02.878424 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:02.878429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:02.879629 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:02.879640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:02.880504 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:02.880513 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:02.880743 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:02.881445 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:02.881469 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:02.883941 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:02.888134 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:02.888142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:02.888162 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:02.888591 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:02.889060 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:02.889066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:02.889076 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:02.889289 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:02.889831 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:02.889839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:02.890197 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 06:34:02.890471 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 06:34:02.891000 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":319,"threads_started":1}
I20260619 06:34:02.893702 555 heartbeater.cc:347] Connected to a master server at 127.28.104.254:36817
I20260619 06:34:02.894022 555 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:02.894748 555 heartbeater.cc:511] Master 127.28.104.254:36817 requested a full tablet report, sending...
I20260619 06:34:02.896292 32698 ts_manager.cc:194] Registered new tserver with Master: 8039d4f121694c088bf65b58ab7299c5 (127.28.104.194:43719)
I20260619 06:34:02.897648 32698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:59843
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:34:02.925596 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36817
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:03.251830 560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:03.252174 560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:03.252286 560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:03.259639 560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:03.259752 560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:03.259863 560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:34:03.270012 560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:03.271257 560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:03.272619 560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:03.285776 567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:03.285787 569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:03.285838 566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:03.286829 560 server_base.cc:1034] running on GCE node
I20260619 06:34:03.287521 560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:03.288489 560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:03.289721 560 hybrid_clock.cc:648] HybridClock initialized: now 1781850843289664 us; error 51 us; skew 500 ppm
I20260619 06:34:03.290062 560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:03 dist-test-slave-nf4n krb5kdc[32653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850843, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:03.293207 560 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:34:03.294708 560 webserver.cc:460] Webserver started at http://127.28.104.195:33969/ using document root <none> and password file <none>
I20260619 06:34:03.295377 560 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:03.295516 560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:03.295828 560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:03.299068 560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "5d6f023af1364ff8b445b1f3a4fb215f"
format_stamp: "Formatted at 2026-06-19 06:34:03 on dist-test-slave-nf4n"
server_key: "e42f24e8314448c39790f94a41df567f"
server_key_iv: "2a3d7096a46914c5b0518ff77f549f99"
server_key_version: "encryptionkey@0"
I20260619 06:34:03.299858 560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "5d6f023af1364ff8b445b1f3a4fb215f"
format_stamp: "Formatted at 2026-06-19 06:34:03 on dist-test-slave-nf4n"
server_key: "e42f24e8314448c39790f94a41df567f"
server_key_iv: "2a3d7096a46914c5b0518ff77f549f99"
server_key_version: "encryptionkey@0"
I20260619 06:34:03.305205 560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 06:34:03.308820 576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:03.310365 560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:03.310536 560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "5d6f023af1364ff8b445b1f3a4fb215f"
format_stamp: "Formatted at 2026-06-19 06:34:03 on dist-test-slave-nf4n"
server_key: "e42f24e8314448c39790f94a41df567f"
server_key_iv: "2a3d7096a46914c5b0518ff77f549f99"
server_key_version: "encryptionkey@0"
I20260619 06:34:03.310738 560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:03.326828 560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:03.331775 560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:03.332047 560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:03.333124 560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:03.334656 560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:03.334738 560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:03.334839 560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:03.334893 560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:03.371825 560 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:34279
I20260619 06:34:03.371881 688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:34279 every 8 connection(s)
I20260619 06:34:03.373407 560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:34:03.375286 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 560
I20260619 06:34:03.375478 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:34:03.376049 29091 external_mini_cluster.cc:1428] Setting key ce050ec21b6e62e9bdbad3606bf57c55
Jun 19 06:34:03 dist-test-slave-nf4n krb5kdc[32653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850843, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:03.390317 420 heartbeater.cc:503] Master 127.28.104.254:36817 was elected leader, sending a full tablet report...
I20260619 06:34:03.394744 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:03.376053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:56993 (local address 127.28.104.254:36817)
0619 06:34:03.376272 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:03.376280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:03.377458 (+ 1178us) server_negotiation.cc:407] Connection header received
0619 06:34:03.379311 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:03.379316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:03.379410 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:03.379497 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:03.380903 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.381783 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.383319 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.383773 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.383872 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:03.387616 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:03.387622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:03.387625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:03.387665 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:03.391590 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.392425 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.392429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.392432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.392578 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.393080 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.393085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.393088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.393458 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:03.393587 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:03.394263 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 06:34:03.394436 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:34:03.395644 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:03.376483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:36817 (local address 127.28.104.195:56993)
0619 06:34:03.377298 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:03.377350 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:03.379055 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:03.379692 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:03.379700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:03.380182 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:03.380748 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:03.380766 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.381913 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.381916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:03.383184 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:03.383196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.383960 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.383965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:03.384180 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:03.384838 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:03.384859 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:03.387391 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:03.391781 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:03.391791 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:03.391807 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:03.392297 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:03.392677 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:03.392680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:03.392683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:03.392858 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:03.393578 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:03.393584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:03.393909 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 06:34:03.394127 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 06:34:03.394511 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":325,"threads_started":1}
I20260619 06:34:03.396994 689 heartbeater.cc:347] Connected to a master server at 127.28.104.254:36817
I20260619 06:34:03.397313 689 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:03.398022 689 heartbeater.cc:511] Master 127.28.104.254:36817 requested a full tablet report, sending...
I20260619 06:34:03.399704 32698 ts_manager.cc:194] Registered new tserver with Master: 5d6f023af1364ff8b445b1f3a4fb215f (127.28.104.195:34279)
I20260619 06:34:03.400537 32698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:56993
I20260619 06:34:03.404263 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:34:03 dist-test-slave-nf4n krb5kdc[32653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:34:03 dist-test-slave-nf4n krb5kdc[32653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:34:03.455036 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:03.438143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.28.104.193:37053)
0619 06:34:03.438829 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:03.438841 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:03.438959 (+ 118us) server_negotiation.cc:407] Connection header received
0619 06:34:03.439260 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:03.439269 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:03.439505 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:03.439689 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:03.441369 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.442484 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.443861 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.444563 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.444724 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:03.449137 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:03.449150 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:03.449172 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:03.449209 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:03.452132 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.452833 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.452842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.452844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.453006 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.453413 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.453417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.453419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.453781 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:03.454016 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:03.454296 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 06:34:03.454563 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":307,"threads_started":1}
Jun 19 06:34:03 dist-test-slave-nf4n krb5kdc[32653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:03.472469 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:03.459240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.28.104.254:36817)
0619 06:34:03.459468 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:03.459474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:03.459574 (+ 100us) server_negotiation.cc:407] Connection header received
0619 06:34:03.459702 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:03.459706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:03.459769 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:03.459854 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:03.461105 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.461822 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.463034 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.463423 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.463512 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:03.467404 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:03.467409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:03.467412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:03.467445 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:03.470007 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.470616 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.470619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.470622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.470743 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.471059 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.471063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.471064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.471471 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:03.471664 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:03.471936 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 06:34:03.472133 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 06:34:03.473008 32698 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50062
I20260619 06:34:03.474084 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 32759
I20260619 06:34:03.486901 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:03.475078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.28.104.193:37053)
0619 06:34:03.475290 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:03.475296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:03.475419 (+ 123us) server_negotiation.cc:407] Connection header received
0619 06:34:03.475541 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:03.475545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:03.475631 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:03.475711 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:03.476833 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.477587 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.478772 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.479186 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.479300 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:03.481486 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:03.481491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:03.481495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:03.481529 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:03.484451 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.485177 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.485182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.485185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.485324 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.485646 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.485650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.485654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.485933 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:03.486219 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:03.486413 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 06:34:03.486591 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:34:03.487618 415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:03.570911 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 32759
I20260619 06:34:03.587774 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 426
Jun 19 06:34:03 dist-test-slave-nf4n krb5kdc[32653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:34:03.604024 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:03.588778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.28.104.194:43719)
0619 06:34:03.589383 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:03.589391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:03.589415 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:34:03.589472 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:03.589478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:03.589690 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:03.589873 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:03.591096 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.592161 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.593506 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.594072 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.594216 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:03.598288 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:03.598308 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:03.598323 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:03.598370 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:03.601211 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.601884 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.601897 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.601900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.602031 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.602416 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.602422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.602425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.602771 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:03.603099 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:03.603406 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 06:34:03.603678 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":328,"threads_started":1}
I20260619 06:34:03.605553 550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:03.691349 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 426
I20260619 06:34:03.708247 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 560
Jun 19 06:34:03 dist-test-slave-nf4n krb5kdc[32653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:34:03.732246 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:03.709252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.28.104.195:34279)
0619 06:34:03.710048 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:03.710056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:03.710089 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:34:03.710155 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:03.710163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:03.710414 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:03.710571 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:03.711774 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.712942 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.714852 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:03.715471 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:03.715588 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:03.726384 (+ 10796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:03.726398 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:03.726413 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:03.726455 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:03.729398 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.730088 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.730096 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.730099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.730218 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:03.730611 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:03.730616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:03.730619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:03.730976 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:03.731309 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:03.731635 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 06:34:03.731916 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":387,"threads_started":1}
I20260619 06:34:03.733609 684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:03.816568 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 560
I20260619 06:34:03.833354 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 32666
I20260619 06:34:03.834587 32728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:03.937383 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 32666
2026-06-19T06:34:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3934 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:34:04 dist-test-slave-nf4n krb5kdc[708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:34:04 dist-test-slave-nf4n krb5kdc[708](info): set up 2 sockets
Jun 19 06:34:04 dist-test-slave-nf4n krb5kdc[708](info): commencing operation
krb5kdc: starting...
W20260619 06:34:05.041368 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:34:05 dist-test-slave-nf4n krb5kdc[708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:34:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:34:05.244303 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:45535
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:45535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:05.569286 722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:05.569659 722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:05.569846 722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:05.577450 722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:05.577559 722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:05.577605 722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:05.577663 722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:05.577711 722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:05.587677 722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:45535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:45535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:05.589007 722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:05.590318 722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:05.601881 727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:05.601984 728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:05.602025 730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:05.602849 722 server_base.cc:1034] running on GCE node
I20260619 06:34:05.603798 722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:05.605894 722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:05.607149 722 hybrid_clock.cc:648] HybridClock initialized: now 1781850845607070 us; error 59 us; skew 500 ppm
I20260619 06:34:05.607578 722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:05 dist-test-slave-nf4n krb5kdc[708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:05.613209 722 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:34:05.615306 722 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:05.615453 722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:05.615856 722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:05.620949 722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "0d9bb04873c941cd83dd1ae3bc07367f"
format_stamp: "Formatted at 2026-06-19 06:34:05 on dist-test-slave-nf4n"
server_key: "696bf9b82cad523eadd1498c1df24f89"
server_key_iv: "115ede397dc0bc11b80c97da74dd4858"
server_key_version: "encryptionkey@0"
I20260619 06:34:05.622300 722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "0d9bb04873c941cd83dd1ae3bc07367f"
format_stamp: "Formatted at 2026-06-19 06:34:05 on dist-test-slave-nf4n"
server_key: "696bf9b82cad523eadd1498c1df24f89"
server_key_iv: "115ede397dc0bc11b80c97da74dd4858"
server_key_version: "encryptionkey@0"
I20260619 06:34:05.630213 722 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 06:34:05.635543 736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:05.637483 722 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260619 06:34:05.637677 722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "0d9bb04873c941cd83dd1ae3bc07367f"
format_stamp: "Formatted at 2026-06-19 06:34:05 on dist-test-slave-nf4n"
server_key: "696bf9b82cad523eadd1498c1df24f89"
server_key_iv: "115ede397dc0bc11b80c97da74dd4858"
server_key_version: "encryptionkey@0"
I20260619 06:34:05.637979 722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:05.682400 722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:05.687033 722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:05.687319 722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:05.705742 722 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:45535
I20260619 06:34:05.705754 787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:45535 every 8 connection(s)
I20260619 06:34:05.706719 722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:05.710419 788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:05.714237 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 722
I20260619 06:34:05.714454 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:05.715291 29091 external_mini_cluster.cc:1428] Setting key 4341d3920687781487fb63a637d865a3
I20260619 06:34:05.724443 788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f: Bootstrap starting.
I20260619 06:34:05.728520 788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:05.729972 788 log.cc:824] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:34:05 dist-test-slave-nf4n krb5kdc[708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:05.733153 788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f: No bootstrap required, opened a new log
I20260619 06:34:05.739140 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:05.717528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.28.104.254:45535)
0619 06:34:05.718329 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:05.718341 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:05.718383 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:34:05.720000 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:05.720023 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:05.720457 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:05.720808 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:05.722745 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:05.724084 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:05.725950 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:05.726673 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:05.726813 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:05.731834 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:05.731842 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:05.731857 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:05.731892 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:05.735076 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:05.735779 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:05.735786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:05.735792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:05.735963 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:05.736353 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:05.736357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:05.736359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:05.736885 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:05.737277 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:05.737680 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 06:34:05.738101 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":383,"threads_started":1}
I20260619 06:34:05.741282 788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9bb04873c941cd83dd1ae3bc07367f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45535 } }
I20260619 06:34:05.741729 788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:05.741842 788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9bb04873c941cd83dd1ae3bc07367f, State: Initialized, Role: FOLLOWER
I20260619 06:34:05.742478 788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9bb04873c941cd83dd1ae3bc07367f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45535 } }
I20260619 06:34:05.742704 788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:05.742873 788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:05.743053 788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:05.745585 788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9bb04873c941cd83dd1ae3bc07367f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45535 } }
I20260619 06:34:05.746081 788 leader_election.cc:302] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d9bb04873c941cd83dd1ae3bc07367f; no voters:
I20260619 06:34:05.746730 788 leader_election.cc:288] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:05.746922 793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:05.748196 793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [term 1 LEADER]: Becoming Leader. State: Replica: 0d9bb04873c941cd83dd1ae3bc07367f, State: Running, Role: LEADER
I20260619 06:34:05.748677 793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9bb04873c941cd83dd1ae3bc07367f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45535 } }
I20260619 06:34:05.748859 788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:05.751808 794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d9bb04873c941cd83dd1ae3bc07367f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9bb04873c941cd83dd1ae3bc07367f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45535 } } }
I20260619 06:34:05.751806 795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d9bb04873c941cd83dd1ae3bc07367f. Latest consensus state: current_term: 1 leader_uuid: "0d9bb04873c941cd83dd1ae3bc07367f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9bb04873c941cd83dd1ae3bc07367f" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45535 } } }
I20260619 06:34:05.752352 794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:05.752358 795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:05.754498 801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:05.760092 801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:05.769411 801 catalog_manager.cc:1347] Generated new cluster ID: ca6694a7e000432fae90f3e68b28f9db
I20260619 06:34:05.769533 801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:05.777051 801 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:05.779122 801 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:05.792269 801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f: Generated new TSK 0
I20260619 06:34:05.793253 801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:34:05.841217 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45535
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:06.186669 814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:06.186975 814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:06.187179 814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:06.195263 814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:06.195405 814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:06.195603 814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:34:06.205616 814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:06.206964 814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:06.208406 814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:06.221819 820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:06.221848 819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:06.221885 822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:06.222535 814 server_base.cc:1034] running on GCE node
I20260619 06:34:06.223132 814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:06.224076 814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:06.225274 814 hybrid_clock.cc:648] HybridClock initialized: now 1781850846225214 us; error 47 us; skew 500 ppm
I20260619 06:34:06.225603 814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:06 dist-test-slave-nf4n krb5kdc[708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850846, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:06.228678 814 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:34:06.229931 814 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:06.230042 814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:06.230366 814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:06.233422 814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "05e02cd1d27e427981af058c27735d9c"
format_stamp: "Formatted at 2026-06-19 06:34:06 on dist-test-slave-nf4n"
server_key: "23765d0aec095abe77a1fc237ba5d716"
server_key_iv: "5af770c89d44f4980b55d84c65d709c7"
server_key_version: "encryptionkey@0"
I20260619 06:34:06.234230 814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "05e02cd1d27e427981af058c27735d9c"
format_stamp: "Formatted at 2026-06-19 06:34:06 on dist-test-slave-nf4n"
server_key: "23765d0aec095abe77a1fc237ba5d716"
server_key_iv: "5af770c89d44f4980b55d84c65d709c7"
server_key_version: "encryptionkey@0"
I20260619 06:34:06.240134 814 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 06:34:06.244068 828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:06.245554 814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:06.245736 814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "05e02cd1d27e427981af058c27735d9c"
format_stamp: "Formatted at 2026-06-19 06:34:06 on dist-test-slave-nf4n"
server_key: "23765d0aec095abe77a1fc237ba5d716"
server_key_iv: "5af770c89d44f4980b55d84c65d709c7"
server_key_version: "encryptionkey@0"
I20260619 06:34:06.245960 814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:06.272362 814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:06.276471 814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:06.276767 814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:06.277822 814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:06.279155 814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:06.279279 814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:06.279394 814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:06.279480 814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:06.312704 814 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:46435
I20260619 06:34:06.312786 941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:46435 every 8 connection(s)
I20260619 06:34:06.313607 814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:34:06.321846 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 814
I20260619 06:34:06.322060 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:34:06.322853 29091 external_mini_cluster.cc:1428] Setting key 095c7720c62370945d8bd609518ffd3c
Jun 19 06:34:06 dist-test-slave-nf4n krb5kdc[708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850846, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:06.335891 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:06.316111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:59665 (local address 127.28.104.254:45535)
0619 06:34:06.316609 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:06.316614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:06.318781 (+ 2167us) server_negotiation.cc:407] Connection header received
0619 06:34:06.319774 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:06.319779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:06.319850 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:06.319938 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:06.321551 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:06.322802 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:06.324990 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:06.325627 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:06.325787 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:06.329484 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:06.329494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:06.329502 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:06.329548 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:06.332439 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:06.333469 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:06.333473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:06.333475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:06.333592 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:06.334068 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:06.334072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:06.334073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:06.334331 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:06.334437 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:06.335389 (+ 952us) server_negotiation.cc:299] Negotiation successful
0619 06:34:06.335626 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":289,"threads_started":1}
I20260619 06:34:06.337460 945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:06.316665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45535 (local address 127.28.104.193:59665)
0619 06:34:06.317466 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:06.317530 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:06.319498 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:06.320145 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:06.320154 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:06.320705 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:06.321368 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:06.321387 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:06.322995 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:06.323001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:06.324837 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:06.324852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:06.325831 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:06.325836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:06.326025 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:06.326751 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:06.326773 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:06.329255 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:06.332585 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:06.332598 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:06.332620 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:06.333346 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:06.333700 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:06.333708 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:06.333714 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:06.333959 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:06.334448 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:06.334459 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:06.334936 (+ 477us) client_negotiation.cc:769] Sending connection context
0619 06:34:06.335270 (+ 334us) client_negotiation.cc:240] Negotiation successful
0619 06:34:06.335870 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":324,"threads_started":1}
I20260619 06:34:06.339071 942 heartbeater.cc:347] Connected to a master server at 127.28.104.254:45535
I20260619 06:34:06.339421 942 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:06.340103 942 heartbeater.cc:511] Master 127.28.104.254:45535 requested a full tablet report, sending...
I20260619 06:34:06.342228 753 ts_manager.cc:194] Registered new tserver with Master: 05e02cd1d27e427981af058c27735d9c (127.28.104.193:46435)
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
I20260619 06:34:06.343921 753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:59665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:34:06.368041 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45535
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:06.691282 948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:06.691641 948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:06.691815 948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:06.699150 948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:06.699309 948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:06.699498 948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:34:06.709419 948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:06.710726 948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:06.712049 948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:06.724939 956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:06.724913 953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:06.724916 954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:06.725518 948 server_base.cc:1034] running on GCE node
I20260619 06:34:06.726166 948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:06.727113 948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:06.728380 948 hybrid_clock.cc:648] HybridClock initialized: now 1781850846728309 us; error 55 us; skew 500 ppm
I20260619 06:34:06.728776 948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:06 dist-test-slave-nf4n krb5kdc[708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850846, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:06.731966 948 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:34:06.733242 948 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:06.733373 948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:06.733702 948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:06.736828 948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "94ff9f4790d443dba79e78f03971d721"
format_stamp: "Formatted at 2026-06-19 06:34:06 on dist-test-slave-nf4n"
server_key: "127e778ab43c72b3538a2ad52f26eb8a"
server_key_iv: "87a43e29b32010dfb5ab8dfab9d62c59"
server_key_version: "encryptionkey@0"
I20260619 06:34:06.737712 948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "94ff9f4790d443dba79e78f03971d721"
format_stamp: "Formatted at 2026-06-19 06:34:06 on dist-test-slave-nf4n"
server_key: "127e778ab43c72b3538a2ad52f26eb8a"
server_key_iv: "87a43e29b32010dfb5ab8dfab9d62c59"
server_key_version: "encryptionkey@0"
I20260619 06:34:06.742861 948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:34:06.746920 962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:06.748380 948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:34:06.748626 948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "94ff9f4790d443dba79e78f03971d721"
format_stamp: "Formatted at 2026-06-19 06:34:06 on dist-test-slave-nf4n"
server_key: "127e778ab43c72b3538a2ad52f26eb8a"
server_key_iv: "87a43e29b32010dfb5ab8dfab9d62c59"
server_key_version: "encryptionkey@0"
I20260619 06:34:06.748909 948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:06.771168 948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:06.776149 948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:06.776511 948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:06.777930 948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:06.779793 948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:06.779903 948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:06.780012 948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:06.780088 948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:06.814530 948 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:40483
I20260619 06:34:06.814580 1075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:40483 every 8 connection(s)
I20260619 06:34:06.815569 948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:34:06.817467 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 948
I20260619 06:34:06.817644 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:34:06.818133 29091 external_mini_cluster.cc:1428] Setting key 38545da09e16589979a000ff050cc1a0
Jun 19 06:34:06 dist-test-slave-nf4n krb5kdc[708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850846, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
I20260619 06:34:06.840493 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:06.818165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:57291 (local address 127.28.104.254:45535)
0619 06:34:06.818386 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:06.818392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:06.819940 (+ 1548us) server_negotiation.cc:407] Connection header received
0619 06:34:06.822013 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:06.822022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:06.822109 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:06.822223 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:06.823857 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:06.825047 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:06.828059 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:06.828495 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:06.828592 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:06.833167 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:06.833173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:06.833177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:06.833211 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:06.836734 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:06.837592 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:06.837601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:06.837606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:06.837808 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:06.838399 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:06.838407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:06.838414 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:06.838866 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:06.839057 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:06.839889 (+ 832us) server_negotiation.cc:299] Negotiation successful
0619 06:34:06.840218 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:34:06.841344 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:06.818780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45535 (local address 127.28.104.194:57291)
0619 06:34:06.819778 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:06.819826 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:06.821732 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:06.822444 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:06.822453 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:06.823024 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:06.823645 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:06.823664 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:06.825183 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:06.825188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:06.827870 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:06.827892 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:06.828748 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:06.828759 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:06.829211 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:06.830008 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:06.830046 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:06.832890 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:06.836898 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:06.836906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:06.836920 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:06.837457 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:06.838025 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:06.838029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:06.838031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:06.838219 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:06.839005 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:06.839011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:06.839376 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 06:34:06.839617 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 06:34:06.840096 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":424,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
I20260619 06:34:06.843036 1076 heartbeater.cc:347] Connected to a master server at 127.28.104.254:45535
I20260619 06:34:06.843390 1076 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:06.844275 1076 heartbeater.cc:511] Master 127.28.104.254:45535 requested a full tablet report, sending...
I20260619 06:34:06.846001 753 ts_manager.cc:194] Registered new tserver with Master: 94ff9f4790d443dba79e78f03971d721 (127.28.104.194:40483)
I20260619 06:34:06.846925 753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:57291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:34:06.865924 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45535
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:07.201633 1081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:07.201995 1081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:07.202175 1081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:07.209896 1081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:07.210093 1081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:07.210307 1081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:34:07.220678 1081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:07.222119 1081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:07.223522 1081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:07.237316 1086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:07.237457 1089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:07.237457 1087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:07.238394 1081 server_base.cc:1034] running on GCE node
I20260619 06:34:07.239025 1081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:07.240026 1081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:07.241281 1081 hybrid_clock.cc:648] HybridClock initialized: now 1781850847241232 us; error 70 us; skew 500 ppm
I20260619 06:34:07.241735 1081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:07 dist-test-slave-nf4n krb5kdc[708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850847, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:07.245913 1081 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:34:07.247128 1081 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:07.247313 1081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:07.247659 1081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:07.251341 1081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "b54102c777d84f3484fb5a2c3dc92074"
format_stamp: "Formatted at 2026-06-19 06:34:07 on dist-test-slave-nf4n"
server_key: "0ad254e243771bd6500bde97f066cf13"
server_key_iv: "e1dc7d2f15b86a301b2d54dc86e6ac5d"
server_key_version: "encryptionkey@0"
I20260619 06:34:07.252182 1081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "b54102c777d84f3484fb5a2c3dc92074"
format_stamp: "Formatted at 2026-06-19 06:34:07 on dist-test-slave-nf4n"
server_key: "0ad254e243771bd6500bde97f066cf13"
server_key_iv: "e1dc7d2f15b86a301b2d54dc86e6ac5d"
server_key_version: "encryptionkey@0"
I20260619 06:34:07.257779 1081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260619 06:34:07.261514 1095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:07.262956 1081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:34:07.263144 1081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "b54102c777d84f3484fb5a2c3dc92074"
format_stamp: "Formatted at 2026-06-19 06:34:07 on dist-test-slave-nf4n"
server_key: "0ad254e243771bd6500bde97f066cf13"
server_key_iv: "e1dc7d2f15b86a301b2d54dc86e6ac5d"
server_key_version: "encryptionkey@0"
I20260619 06:34:07.263391 1081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:07.279747 1081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:07.284178 1081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:07.284464 1081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:07.285451 1081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:07.286779 1081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:07.286899 1081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:07.287029 1081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:07.287112 1081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:07.321266 1081 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:37701
I20260619 06:34:07.321309 1207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:37701 every 8 connection(s)
I20260619 06:34:07.322149 1081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:34:07.325743 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 1081
I20260619 06:34:07.325937 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:34:07.326507 29091 external_mini_cluster.cc:1428] Setting key 20f87ec8695d31fc7a21f4bdda4ce539
Jun 19 06:34:07 dist-test-slave-nf4n krb5kdc[708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850847, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:07.344033 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.324581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:50815 (local address 127.28.104.254:45535)
0619 06:34:07.324791 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.324795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.326441 (+ 1646us) server_negotiation.cc:407] Connection header received
0619 06:34:07.328822 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.328826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.328890 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.328970 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:07.330867 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.331609 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.333349 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.333771 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.333881 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.337491 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:07.337496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:07.337499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:07.337533 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:07.340793 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:07.341636 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:07.341640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:07.341642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:07.341766 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:07.342211 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:07.342215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:07.342217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:07.342579 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:07.342736 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.343453 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.343681 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:34:07.345140 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.325166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45535 (local address 127.28.104.195:50815)
0619 06:34:07.326238 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:07.326293 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:07.328424 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:07.329337 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:07.329350 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:07.330007 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:07.330676 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.330703 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.331790 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.331798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.333204 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.333217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.334015 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.334020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.334256 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.334949 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:07.334973 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:07.337279 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:07.340957 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:07.340968 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:07.340991 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:07.341503 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:07.341901 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:07.341910 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:07.341913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:07.342090 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:07.342705 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:07.342713 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:07.343078 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 06:34:07.343341 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 06:34:07.343908 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":395,"threads_started":1}
I20260619 06:34:07.346557 1208 heartbeater.cc:347] Connected to a master server at 127.28.104.254:45535
I20260619 06:34:07.346999 1208 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:07.347678 942 heartbeater.cc:503] Master 127.28.104.254:45535 was elected leader, sending a full tablet report...
I20260619 06:34:07.347725 1208 heartbeater.cc:511] Master 127.28.104.254:45535 requested a full tablet report, sending...
I20260619 06:34:07.349732 753 ts_manager.cc:194] Registered new tserver with Master: b54102c777d84f3484fb5a2c3dc92074 (127.28.104.195:37701)
I20260619 06:34:07.350629 753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:50815
I20260619 06:34:07.354825 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:34:07.372645 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.359299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.28.104.254:45535)
0619 06:34:07.359493 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.359501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.360037 (+ 536us) server_negotiation.cc:407] Connection header received
0619 06:34:07.360161 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.360165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.360231 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.360318 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:07.361822 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.362518 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.363867 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.364302 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.364399 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.366726 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:07.366732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:07.366736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:07.366775 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:07.370083 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:07.370707 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:07.370712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:07.370714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:07.370857 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:07.371202 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:07.371206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:07.371208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:07.371572 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:07.371828 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.372050 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.372236 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:34:07.377398 753 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:34:07.384284 753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:34:07.422389 1222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.397155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.28.104.195:37701)
0619 06:34:07.412854 (+ 15699us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.412869 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.412928 (+ 59us) server_negotiation.cc:407] Connection header received
0619 06:34:07.413073 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.413085 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.413412 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.414875 (+ 1463us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:07.415227 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.416992 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.419665 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.421104 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.421311 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.421503 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.421868 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.422030 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15445,"thread_start_us":16422,"threads_started":1}
I20260619 06:34:07.422490 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.396802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:37701 (local address 127.0.0.1:56390)
0619 06:34:07.398625 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:07.398676 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:07.398903 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:07.413921 (+ 15018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:07.413933 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:07.413992 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:07.414567 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.414585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.417146 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.417151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.419511 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.419522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.421325 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.421331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.421616 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.421655 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:34:07.421812 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 06:34:07.422026 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"mutex_wait_us":25,"thread_start_us":19457,"threads_started":1}
I20260619 06:34:07.425540 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.397880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.28.104.193:46435)
0619 06:34:07.415501 (+ 17621us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.415514 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.416236 (+ 722us) server_negotiation.cc:407] Connection header received
0619 06:34:07.417013 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.417020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.417310 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.417518 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:07.418835 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.420422 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.423490 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.424435 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.424661 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.424854 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.425010 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.425213 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17380,"thread_start_us":15470,"threads_started":1}
I20260619 06:34:07.425555 1224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.397157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:46435 (local address 127.0.0.1:44790)
0619 06:34:07.416118 (+ 18961us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:07.416139 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:07.416836 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:07.417514 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:07.417519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:07.417543 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:07.418558 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.418571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.420574 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.420578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.423335 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.423350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.424664 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.424670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.424826 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.424856 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:34:07.424970 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 06:34:07.425124 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18820,"mutex_wait_us":106,"thread_start_us":977,"threads_started":1}
I20260619 06:34:07.427299 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.400259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.28.104.194:40483)
0619 06:34:07.416864 (+ 16605us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.416870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.416892 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:34:07.417011 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.417019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.417311 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.417517 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:07.418721 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.420398 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.424749 (+ 4351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.426198 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.426377 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.426533 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.426808 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.426983 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16435,"thread_start_us":16283,"threads_started":1}
I20260619 06:34:07.427086 1143 tablet_service.cc:1467] Processing CreateTablet for tablet 394526e2fbd94a5b837c7e2d5d5eaa77 (DEFAULT_TABLE table=test-table [id=8050a991acea4c5facca5f74a210aaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:07.428112 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.400059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40483 (local address 127.0.0.1:52846)
0619 06:34:07.416544 (+ 16485us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:07.416570 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:07.416693 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:07.417780 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:07.417783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:07.417804 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:07.418544 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.418554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.420572 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.420577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.424525 (+ 3948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.424538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.426422 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.426429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.426593 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.426635 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 06:34:07.426738 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:34:07.427768 (+ 1030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16343,"mutex_wait_us":1448,"thread_start_us":15758,"threads_started":1}
I20260619 06:34:07.428622 1143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 394526e2fbd94a5b837c7e2d5d5eaa77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:07.429337 877 tablet_service.cc:1467] Processing CreateTablet for tablet 394526e2fbd94a5b837c7e2d5d5eaa77 (DEFAULT_TABLE table=test-table [id=8050a991acea4c5facca5f74a210aaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:07.430377 877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 394526e2fbd94a5b837c7e2d5d5eaa77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:07.438648 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 394526e2fbd94a5b837c7e2d5d5eaa77 (DEFAULT_TABLE table=test-table [id=8050a991acea4c5facca5f74a210aaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:07.440129 1010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 394526e2fbd94a5b837c7e2d5d5eaa77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:07.440943 1226 tablet_bootstrap.cc:492] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: Bootstrap starting.
I20260619 06:34:07.444357 1226 tablet_bootstrap.cc:654] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:07.445678 1226 log.cc:824] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:07.445776 1227 tablet_bootstrap.cc:492] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: Bootstrap starting.
I20260619 06:34:07.449501 1227 tablet_bootstrap.cc:654] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:07.449507 1226 tablet_bootstrap.cc:492] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: No bootstrap required, opened a new log
I20260619 06:34:07.449856 1226 ts_tablet_manager.cc:1397] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 06:34:07.450654 1227 log.cc:824] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:07.453349 1227 tablet_bootstrap.cc:492] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: No bootstrap required, opened a new log
I20260619 06:34:07.453621 1227 ts_tablet_manager.cc:1397] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 06:34:07.456070 1230 tablet_bootstrap.cc:492] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: Bootstrap starting.
I20260619 06:34:07.459453 1226 raft_consensus.cc:348] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54102c777d84f3484fb5a2c3dc92074" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 37701 } } peers { permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 } } peers { permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 } }
I20260619 06:34:07.459774 1226 raft_consensus.cc:374] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:07.459842 1226 raft_consensus.cc:729] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e02cd1d27e427981af058c27735d9c, State: Initialized, Role: FOLLOWER
I20260619 06:34:07.460440 1226 consensus_queue.cc:260] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54102c777d84f3484fb5a2c3dc92074" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 37701 } } peers { permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 } } peers { permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 } }
I20260619 06:34:07.460773 1230 tablet_bootstrap.cc:654] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:07.462191 1230 log.cc:824] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:07.462520 1226 ts_tablet_manager.cc:1428] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260619 06:34:07.463593 1227 raft_consensus.cc:348] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54102c777d84f3484fb5a2c3dc92074" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 37701 } } peers { permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 } } peers { permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 } }
I20260619 06:34:07.464037 1227 raft_consensus.cc:374] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:07.464149 1227 raft_consensus.cc:729] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54102c777d84f3484fb5a2c3dc92074, State: Initialized, Role: FOLLOWER
I20260619 06:34:07.464742 1227 consensus_queue.cc:260] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54102c777d84f3484fb5a2c3dc92074" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 37701 } } peers { permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 } } peers { permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 } }
I20260619 06:34:07.465916 1208 heartbeater.cc:503] Master 127.28.104.254:45535 was elected leader, sending a full tablet report...
I20260619 06:34:07.466385 1230 tablet_bootstrap.cc:492] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: No bootstrap required, opened a new log
I20260619 06:34:07.466521 1227 ts_tablet_manager.cc:1428] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260619 06:34:07.466749 1230 ts_tablet_manager.cc:1397] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260619 06:34:07.475018 1230 raft_consensus.cc:348] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54102c777d84f3484fb5a2c3dc92074" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 37701 } } peers { permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 } } peers { permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 } }
I20260619 06:34:07.475394 1230 raft_consensus.cc:374] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:07.475507 1230 raft_consensus.cc:729] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ff9f4790d443dba79e78f03971d721, State: Initialized, Role: FOLLOWER
I20260619 06:34:07.476156 1230 consensus_queue.cc:260] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54102c777d84f3484fb5a2c3dc92074" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 37701 } } peers { permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 } } peers { permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 } }
I20260619 06:34:07.477324 1076 heartbeater.cc:503] Master 127.28.104.254:45535 was elected leader, sending a full tablet report...
I20260619 06:34:07.477831 1230 ts_tablet_manager.cc:1428] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 06:34:07.523964 1233 raft_consensus.cc:482] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:34:07.524344 1233 raft_consensus.cc:504] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54102c777d84f3484fb5a2c3dc92074" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 37701 } } peers { permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 } } peers { permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 } }
I20260619 06:34:07.526263 1233 leader_election.cc:288] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05e02cd1d27e427981af058c27735d9c (127.28.104.193:46435), 94ff9f4790d443dba79e78f03971d721 (127.28.104.194:40483)
I20260619 06:34:07.533207 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.526760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:51469 (local address 127.28.104.193:46435)
0619 06:34:07.526979 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.526986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.527119 (+ 133us) server_negotiation.cc:407] Connection header received
0619 06:34:07.527400 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.527407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.527499 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.527622 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:07.528049 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.529250 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.531079 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.532295 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.532426 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.532484 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.532798 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.532894 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:34:07.533257 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.526592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:46435 (local address 127.28.104.195:51469)
0619 06:34:07.527016 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:07.527042 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:07.527212 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:07.527635 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:07.527638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:07.527664 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:07.527881 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.527890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.529392 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.529398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.530911 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.530922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.532443 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.532450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.532636 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.532672 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:34:07.532771 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:34:07.532934 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1}
I20260619 06:34:07.534165 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.526901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:39215 (local address 127.28.104.194:40483)
0619 06:34:07.527099 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.527107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.527791 (+ 684us) server_negotiation.cc:407] Connection header received
0619 06:34:07.527996 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.528004 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.528072 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.528139 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:07.528918 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.529886 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.532076 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.533208 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.533373 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.533448 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.533895 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.533990 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:34:07.534543 1235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.526627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40483 (local address 127.28.104.195:39215)
0619 06:34:07.527666 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:07.527685 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:07.527858 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:07.528298 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:07.528303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:07.528332 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:07.528741 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.528754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.530064 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.530072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.531909 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:07.531926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.533573 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.533580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:07.533731 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.533755 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:34:07.533840 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:34:07.533991 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":25,"thread_start_us":657,"threads_started":1}
I20260619 06:34:07.534498 897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394526e2fbd94a5b837c7e2d5d5eaa77" candidate_uuid: "b54102c777d84f3484fb5a2c3dc92074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e02cd1d27e427981af058c27735d9c" is_pre_election: true
I20260619 06:34:07.535239 897 raft_consensus.cc:2454] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b54102c777d84f3484fb5a2c3dc92074 in term 0.
I20260619 06:34:07.535768 1031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394526e2fbd94a5b837c7e2d5d5eaa77" candidate_uuid: "b54102c777d84f3484fb5a2c3dc92074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ff9f4790d443dba79e78f03971d721" is_pre_election: true
I20260619 06:34:07.536022 1097 leader_election.cc:302] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e02cd1d27e427981af058c27735d9c, b54102c777d84f3484fb5a2c3dc92074; no voters:
I20260619 06:34:07.536324 1031 raft_consensus.cc:2454] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b54102c777d84f3484fb5a2c3dc92074 in term 0.
I20260619 06:34:07.536442 1233 raft_consensus.cc:2781] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:34:07.536620 1233 raft_consensus.cc:482] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:34:07.536758 1233 raft_consensus.cc:3037] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:07.539346 1233 raft_consensus.cc:504] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54102c777d84f3484fb5a2c3dc92074" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 37701 } } peers { permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 } } peers { permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 } }
I20260619 06:34:07.539992 1233 leader_election.cc:288] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [CANDIDATE]: Term 1 election: Requested vote from peers 05e02cd1d27e427981af058c27735d9c (127.28.104.193:46435), 94ff9f4790d443dba79e78f03971d721 (127.28.104.194:40483)
I20260619 06:34:07.540434 897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394526e2fbd94a5b837c7e2d5d5eaa77" candidate_uuid: "b54102c777d84f3484fb5a2c3dc92074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e02cd1d27e427981af058c27735d9c"
I20260619 06:34:07.540448 1031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394526e2fbd94a5b837c7e2d5d5eaa77" candidate_uuid: "b54102c777d84f3484fb5a2c3dc92074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ff9f4790d443dba79e78f03971d721"
I20260619 06:34:07.540658 1031 raft_consensus.cc:3037] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:07.540652 897 raft_consensus.cc:3037] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:07.542877 1031 raft_consensus.cc:2454] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b54102c777d84f3484fb5a2c3dc92074 in term 1.
I20260619 06:34:07.543088 897 raft_consensus.cc:2454] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b54102c777d84f3484fb5a2c3dc92074 in term 1.
I20260619 06:34:07.543354 1096 leader_election.cc:302] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ff9f4790d443dba79e78f03971d721, b54102c777d84f3484fb5a2c3dc92074; no voters:
I20260619 06:34:07.543687 1233 raft_consensus.cc:2781] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:07.544179 1233 raft_consensus.cc:686] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 1 LEADER]: Becoming Leader. State: Replica: b54102c777d84f3484fb5a2c3dc92074, State: Running, Role: LEADER
I20260619 06:34:07.544735 1233 consensus_queue.cc:237] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54102c777d84f3484fb5a2c3dc92074" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 37701 } } peers { permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 } } peers { permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 } }
I20260619 06:34:07.550050 752 catalog_manager.cc:5495] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 reported cstate change: term changed from 0 to 1, leader changed from <none> to b54102c777d84f3484fb5a2c3dc92074 (127.28.104.195). New cstate: current_term: 1 leader_uuid: "b54102c777d84f3484fb5a2c3dc92074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54102c777d84f3484fb5a2c3dc92074" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 37701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 } health_report { overall_health: UNKNOWN } } }
W20260619 06:34:07.565893 943 tablet.cc:2369] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:34:07.567510 1077 tablet.cc:2369] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:34:07.573655 1209 tablet.cc:2369] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:34:07.599902 1222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.595046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.28.104.195:37701)
0619 06:34:07.595278 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.595283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.595301 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:34:07.595388 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.595391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.595467 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.595592 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:34:07.595902 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.596610 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.597964 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.598345 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.598439 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.598854 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:34:07.599365 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:34:07.599484 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.599524 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.599588 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 06:34:07.608218 1031 raft_consensus.cc:1261] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721 [term 1 FOLLOWER]: Refusing update from remote peer b54102c777d84f3484fb5a2c3dc92074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:07.608217 897 raft_consensus.cc:1261] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c [term 1 FOLLOWER]: Refusing update from remote peer b54102c777d84f3484fb5a2c3dc92074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:07.609225 1236 consensus_queue.cc:1035] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e02cd1d27e427981af058c27735d9c" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:07.609547 1233 consensus_queue.cc:1035] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ff9f4790d443dba79e78f03971d721" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:07.618461 1239 mvcc.cc:205] Tried to move back new op lower bound from 7298461071792930816 to 7298461071547654144. Current Snapshot: MvccSnapshot[applied={T|T < 7298461071792930816}]
I20260619 06:34:07.621448 1240 mvcc.cc:205] Tried to move back new op lower bound from 7298461071792930816 to 7298461071547654144. Current Snapshot: MvccSnapshot[applied={T|T < 7298461071792930816}]
I20260619 06:34:07.637740 752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:07.638021 752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:07.641412 752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d9bb04873c941cd83dd1ae3bc07367f: Sending DeleteTablet for 3 replicas of tablet 394526e2fbd94a5b837c7e2d5d5eaa77
I20260619 06:34:07.642475 1143 tablet_service.cc:1514] Processing DeleteTablet for tablet 394526e2fbd94a5b837c7e2d5d5eaa77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:07 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:56390
I20260619 06:34:07.642584 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 394526e2fbd94a5b837c7e2d5d5eaa77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:07 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:44790
I20260619 06:34:07.642584 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 394526e2fbd94a5b837c7e2d5d5eaa77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:07 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:52846
I20260619 06:34:07.646193 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 814
I20260619 06:34:07.659060 1251 tablet_replica.cc:330] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: stopping tablet replica
I20260619 06:34:07.659669 1251 raft_consensus.cc:2229] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:07.661123 1250 tablet_replica.cc:330] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: stopping tablet replica
I20260619 06:34:07.661724 1250 raft_consensus.cc:2229] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:07.662294 1250 raft_consensus.cc:2258] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:07.663872 1251 raft_consensus.cc:2258] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:07.664548 1250 ts_tablet_manager.cc:1905] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:07.667959 1251 ts_tablet_manager.cc:1905] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 06:34:07 dist-test-slave-nf4n krb5kdc[708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:34:07.673894 1250 ts_tablet_manager.cc:1918] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:07.675172 1250 log.cc:1196] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/394526e2fbd94a5b837c7e2d5d5eaa77
I20260619 06:34:07.675797 1250 ts_tablet_manager.cc:1939] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 05e02cd1d27e427981af058c27735d9c: Deleting consensus metadata
I20260619 06:34:07.677872 1251 ts_tablet_manager.cc:1918] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:07.678040 1251 log.cc:1196] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/394526e2fbd94a5b837c7e2d5d5eaa77
I20260619 06:34:07.678593 1251 ts_tablet_manager.cc:1939] T 394526e2fbd94a5b837c7e2d5d5eaa77 P 94ff9f4790d443dba79e78f03971d721: Deleting consensus metadata
I20260619 06:34:07.680557 737 catalog_manager.cc:4841] TS 94ff9f4790d443dba79e78f03971d721 (127.28.104.194:40483): tablet 394526e2fbd94a5b837c7e2d5d5eaa77 (table test-table [id=8050a991acea4c5facca5f74a210aaef]) successfully deleted
I20260619 06:34:07.681447 1249 tablet_replica.cc:330] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: stopping tablet replica
I20260619 06:34:07.681861 738 catalog_manager.cc:4841] TS 05e02cd1d27e427981af058c27735d9c (127.28.104.193:46435): tablet 394526e2fbd94a5b837c7e2d5d5eaa77 (table test-table [id=8050a991acea4c5facca5f74a210aaef]) successfully deleted
I20260619 06:34:07.681962 1249 raft_consensus.cc:2229] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:34:07.682670 1249 raft_consensus.cc:2258] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:07.684908 1249 ts_tablet_manager.cc:1905] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:07.685367 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.647843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.28.104.193:46435)
0619 06:34:07.648074 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.648083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.653351 (+ 5268us) server_negotiation.cc:407] Connection header received
0619 06:34:07.654529 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.654538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.654666 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.654748 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:07.660748 (+ 6000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.661998 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.664514 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.665159 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.665293 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.677222 (+ 11929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:07.677236 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:07.677255 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:07.677299 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:07.681253 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:07.682826 (+ 1573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:07.682836 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:07.682840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:07.683028 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:07.683532 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:07.683538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:07.683541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:07.684027 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:07.684334 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.684698 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.684918 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:34:07.686282 937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:07.693756 1249 ts_tablet_manager.cc:1918] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:07.693946 1249 log.cc:1196] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/394526e2fbd94a5b837c7e2d5d5eaa77
I20260619 06:34:07.694540 1249 ts_tablet_manager.cc:1939] T 394526e2fbd94a5b837c7e2d5d5eaa77 P b54102c777d84f3484fb5a2c3dc92074: Deleting consensus metadata
I20260619 06:34:07.696605 739 catalog_manager.cc:4841] TS b54102c777d84f3484fb5a2c3dc92074 (127.28.104.195:37701): tablet 394526e2fbd94a5b837c7e2d5d5eaa77 (table test-table [id=8050a991acea4c5facca5f74a210aaef]) successfully deleted
I20260619 06:34:07.778563 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 814
I20260619 06:34:07.795933 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 948
Jun 19 06:34:07 dist-test-slave-nf4n krb5kdc[708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:34:07.811151 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.796941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.28.104.194:40483)
0619 06:34:07.797167 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.797171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.797187 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:34:07.797242 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.797246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.797320 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.797430 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:07.798667 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.799431 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.800670 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.801136 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.801263 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.805327 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:07.805345 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:07.805361 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:07.805403 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:07.808206 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:07.808915 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:07.808929 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:07.808931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:07.809125 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:07.809596 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:07.809601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:07.809605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:07.809971 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:07.810274 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.810578 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.810807 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:34:07.811888 1071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:07.901212 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 948
I20260619 06:34:07.918452 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 1081
Jun 19 06:34:07 dist-test-slave-nf4n krb5kdc[708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:34:07.934885 1222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:07.919296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.28.104.195:37701)
0619 06:34:07.919471 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:07.919478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:07.919505 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:34:07.919721 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:07.919731 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:07.919817 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:07.919920 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:07.921504 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.922327 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.923709 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:07.924158 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:07.924296 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:07.928924 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:07.928937 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:07.928952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:07.928998 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:07.932027 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:07.932723 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:07.932731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:07.932733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:07.932883 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:07.933307 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:07.933314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:07.933319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:07.933691 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:07.934032 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:07.934282 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 06:34:07.934510 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:34:07.935655 1203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:08.058816 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 1081
I20260619 06:34:08.078398 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 722
I20260619 06:34:08.079509 783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:08.189270 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 722
2026-06-19T06:34:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4251 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T06:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:08Z Disabled control of system clock
I20260619 06:34:08.257934 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:36201
--webserver_interface=127.28.104.254
--webserver_port=0
--builtin_ntp_servers=127.28.104.212:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:36201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:34:08.580400 1260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:08.580680 1260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:08.580762 1260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:08.588008 1260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:08.588119 1260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:08.588156 1260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:08.588213 1260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:08.588260 1260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:08.597492 1260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:36201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:36201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:08.598740 1260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:08.600440 1260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:08.612375 1266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:08.612540 1265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:08.612402 1268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:08.613269 1260 server_base.cc:1034] running on GCE node
I20260619 06:34:08.614049 1260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:08.615451 1260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:08.616688 1260 hybrid_clock.cc:648] HybridClock initialized: now 1781850848616646 us; error 44 us; skew 500 ppm
I20260619 06:34:08.617142 1260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:08.619395 1260 webserver.cc:460] Webserver started at http://127.28.104.254:35835/ using document root <none> and password file <none>
I20260619 06:34:08.620146 1260 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:08.620271 1260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:08.620601 1260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:08.623572 1260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3"
format_stamp: "Formatted at 2026-06-19 06:34:08 on dist-test-slave-nf4n"
server_key: "0ca0def11104fe3437745af8a7dae63a"
server_key_iv: "e31efc5a3105b497bc4893ad9b5d3eb4"
server_key_version: "encryptionkey@0"
I20260619 06:34:08.624375 1260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3"
format_stamp: "Formatted at 2026-06-19 06:34:08 on dist-test-slave-nf4n"
server_key: "0ca0def11104fe3437745af8a7dae63a"
server_key_iv: "e31efc5a3105b497bc4893ad9b5d3eb4"
server_key_version: "encryptionkey@0"
I20260619 06:34:08.629513 1260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 06:34:08.632879 1274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:08.634195 1260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:34:08.634366 1260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3"
format_stamp: "Formatted at 2026-06-19 06:34:08 on dist-test-slave-nf4n"
server_key: "0ca0def11104fe3437745af8a7dae63a"
server_key_iv: "e31efc5a3105b497bc4893ad9b5d3eb4"
server_key_version: "encryptionkey@0"
I20260619 06:34:08.634575 1260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:08.640242 1260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:08.641113 1260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:08.641404 1260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:08.659827 1260 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:36201
I20260619 06:34:08.659844 1327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:36201 every 8 connection(s)
I20260619 06:34:08.661226 1260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:08.665158 1328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:08.666123 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 1260
I20260619 06:34:08.666379 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:08.667013 29091 external_mini_cluster.cc:1428] Setting key 268af4db3b2ed41e1d5e70d28df0cc10
I20260619 06:34:08.675807 1331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:08.669001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.28.104.254:36201)
0619 06:34:08.670174 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:08.670191 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:08.670265 (+ 74us) server_negotiation.cc:407] Connection header received
0619 06:34:08.671979 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:08.672003 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:08.672460 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:08.672933 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:08.674069 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:08.674075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:08.674088 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:08.674102 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:08.674159 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:08.674175 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:08.674416 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:08.674563 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 06:34:08.674826 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":383,"threads_started":1}
I20260619 06:34:08.678992 1328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3: Bootstrap starting.
I20260619 06:34:08.682045 1328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:08.683255 1328 log.cc:824] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:08.685828 1328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3: No bootstrap required, opened a new log
I20260619 06:34:08.692963 1328 raft_consensus.cc:348] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36201 } }
I20260619 06:34:08.693405 1328 raft_consensus.cc:374] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:08.693507 1328 raft_consensus.cc:729] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af80a1eede3045b2bb4bdf2ec9b07ba3, State: Initialized, Role: FOLLOWER
I20260619 06:34:08.694080 1328 consensus_queue.cc:260] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36201 } }
I20260619 06:34:08.694326 1328 raft_consensus.cc:388] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:08.694466 1328 raft_consensus.cc:482] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:08.694621 1328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:08.696745 1328 raft_consensus.cc:504] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36201 } }
I20260619 06:34:08.697297 1328 leader_election.cc:302] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af80a1eede3045b2bb4bdf2ec9b07ba3; no voters:
I20260619 06:34:08.697989 1328 leader_election.cc:288] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:08.698112 1333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:08.699146 1333 raft_consensus.cc:686] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [term 1 LEADER]: Becoming Leader. State: Replica: af80a1eede3045b2bb4bdf2ec9b07ba3, State: Running, Role: LEADER
I20260619 06:34:08.699725 1333 consensus_queue.cc:237] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36201 } }
I20260619 06:34:08.700371 1328 sys_catalog.cc:564] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:08.702817 1335 sys_catalog.cc:455] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af80a1eede3045b2bb4bdf2ec9b07ba3. Latest consensus state: current_term: 1 leader_uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36201 } } }
I20260619 06:34:08.703176 1335 sys_catalog.cc:458] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:08.703617 1334 sys_catalog.cc:455] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af80a1eede3045b2bb4bdf2ec9b07ba3" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36201 } } }
I20260619 06:34:08.703985 1334 sys_catalog.cc:458] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:08.705293 1340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:08.710932 1340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:08.719803 1340 catalog_manager.cc:1347] Generated new cluster ID: b4ec1b4938fd4349bc36f37ebd9a9ce9
I20260619 06:34:08.719926 1340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:08.740070 1340 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:08.740571 1340 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:08.754005 1340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3: Generated new TSK 0
I20260619 06:34:08.755072 1340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:34:08.761286 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36201
--builtin_ntp_servers=127.28.104.212:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:34:09.097370 1352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:09.097654 1352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:09.097745 1352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:09.104983 1352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:09.105141 1352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:09.105355 1352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:34:09.114593 1352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:09.115862 1352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:09.117921 1352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:09.131047 1360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:09.131179 1357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:09.131124 1358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:09.132025 1352 server_base.cc:1034] running on GCE node
I20260619 06:34:09.132635 1352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:09.133715 1352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:09.134919 1352 hybrid_clock.cc:648] HybridClock initialized: now 1781850849134868 us; error 41 us; skew 500 ppm
I20260619 06:34:09.135248 1352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:09.141930 1352 webserver.cc:460] Webserver started at http://127.28.104.193:44949/ using document root <none> and password file <none>
I20260619 06:34:09.142668 1352 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:09.142792 1352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:09.143128 1352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:09.146088 1352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "2aff50c176844836876e1f936f1edc49"
format_stamp: "Formatted at 2026-06-19 06:34:09 on dist-test-slave-nf4n"
server_key: "8becaf1edce7e18046868657cb566cfe"
server_key_iv: "f523dd64e8d98d42d04d3b42323694f0"
server_key_version: "encryptionkey@0"
I20260619 06:34:09.146911 1352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "2aff50c176844836876e1f936f1edc49"
format_stamp: "Formatted at 2026-06-19 06:34:09 on dist-test-slave-nf4n"
server_key: "8becaf1edce7e18046868657cb566cfe"
server_key_iv: "f523dd64e8d98d42d04d3b42323694f0"
server_key_version: "encryptionkey@0"
I20260619 06:34:09.152091 1352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:34:09.155520 1366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:09.156801 1352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:34:09.156978 1352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "2aff50c176844836876e1f936f1edc49"
format_stamp: "Formatted at 2026-06-19 06:34:09 on dist-test-slave-nf4n"
server_key: "8becaf1edce7e18046868657cb566cfe"
server_key_iv: "f523dd64e8d98d42d04d3b42323694f0"
server_key_version: "encryptionkey@0"
I20260619 06:34:09.157240 1352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:09.162843 1352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:09.163637 1352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:09.163921 1352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:09.164826 1352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:09.166250 1352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:09.166369 1352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:09.166498 1352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:09.166586 1352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:09.200497 1352 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:36871
I20260619 06:34:09.200515 1480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:36871 every 8 connection(s)
I20260619 06:34:09.202113 1352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:34:09.210844 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:09.204919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:34349 (local address 127.28.104.254:36201)
0619 06:34:09.205398 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:09.205402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:09.206844 (+ 1442us) server_negotiation.cc:407] Connection header received
0619 06:34:09.208524 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:09.208529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:09.208595 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:09.208720 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:09.210042 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:09.210048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:09.210053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:09.210071 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:09.210118 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:09.210130 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:09.210249 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:09.210470 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 06:34:09.210537 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":273,"threads_started":1}
I20260619 06:34:09.210881 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 1352
I20260619 06:34:09.211128 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:34:09.211777 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:09.205686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:36201 (local address 127.28.104.193:34349)
0619 06:34:09.206687 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:09.206729 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:09.208246 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:09.208960 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:09.208968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:09.209111 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:09.209748 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:34:09.209751 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:09.209791 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:34:09.209880 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:09.210221 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:09.210231 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:09.210239 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:34:09.210527 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 06:34:09.210783 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":265,"threads_started":1}
I20260619 06:34:09.211788 29091 external_mini_cluster.cc:1428] Setting key a1c68534f6cdcbaa6cacac7de17c46d4
I20260619 06:34:09.213526 1481 heartbeater.cc:347] Connected to a master server at 127.28.104.254:36201
I20260619 06:34:09.213893 1481 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:09.214468 1481 heartbeater.cc:511] Master 127.28.104.254:36201 requested a full tablet report, sending...
I20260619 06:34:09.214840 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36201
--builtin_ntp_servers=127.28.104.212:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 06:34:09.216794 1292 ts_manager.cc:194] Registered new tserver with Master: 2aff50c176844836876e1f936f1edc49 (127.28.104.193:36871)
W20260619 06:34:09.551137 1485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:09.551476 1485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:09.551683 1485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:09.559314 1485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:09.559438 1485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:09.559638 1485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:34:09.569301 1485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:09.570734 1485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:09.572614 1485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:09.586357 1491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:09.586306 1493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:09.586406 1490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:09.587466 1485 server_base.cc:1034] running on GCE node
I20260619 06:34:09.588203 1485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:09.589272 1485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:09.590508 1485 hybrid_clock.cc:648] HybridClock initialized: now 1781850849590462 us; error 37 us; skew 500 ppm
I20260619 06:34:09.590893 1485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:09.593406 1485 webserver.cc:460] Webserver started at http://127.28.104.194:44727/ using document root <none> and password file <none>
I20260619 06:34:09.594177 1485 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:09.594298 1485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:09.594666 1485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:09.598002 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "ed8a5f8308de422b9c62d7f97fd36dce"
format_stamp: "Formatted at 2026-06-19 06:34:09 on dist-test-slave-nf4n"
server_key: "b37d1e95a554bf4a7564544cf5695fc6"
server_key_iv: "485f6f3d02498cf448af5bed5f654559"
server_key_version: "encryptionkey@0"
I20260619 06:34:09.598860 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "ed8a5f8308de422b9c62d7f97fd36dce"
format_stamp: "Formatted at 2026-06-19 06:34:09 on dist-test-slave-nf4n"
server_key: "b37d1e95a554bf4a7564544cf5695fc6"
server_key_iv: "485f6f3d02498cf448af5bed5f654559"
server_key_version: "encryptionkey@0"
I20260619 06:34:09.604306 1485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 06:34:09.608111 1499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:09.609611 1485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 06:34:09.609810 1485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "ed8a5f8308de422b9c62d7f97fd36dce"
format_stamp: "Formatted at 2026-06-19 06:34:09 on dist-test-slave-nf4n"
server_key: "b37d1e95a554bf4a7564544cf5695fc6"
server_key_iv: "485f6f3d02498cf448af5bed5f654559"
server_key_version: "encryptionkey@0"
I20260619 06:34:09.610039 1485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:09.616119 1485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:09.616966 1485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:09.617326 1485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:09.618269 1485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:09.619720 1485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:09.619835 1485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:09.619969 1485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:09.620059 1485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:09.654706 1485 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:42613
I20260619 06:34:09.654799 1611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:42613 every 8 connection(s)
I20260619 06:34:09.656201 1485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:34:09.665993 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:09.659097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:51467 (local address 127.28.104.254:36201)
0619 06:34:09.659359 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:09.659366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:09.661095 (+ 1729us) server_negotiation.cc:407] Connection header received
0619 06:34:09.663464 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:09.663469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:09.663525 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:09.663611 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:09.665171 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:09.665178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:09.665183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:09.665213 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:09.665248 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:09.665262 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:09.665389 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:09.665601 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 06:34:09.665655 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 06:34:09.666144 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 1485
I20260619 06:34:09.666348 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:34:09.666801 29091 external_mini_cluster.cc:1428] Setting key 995734bf8f7e95605f4e7e66df4375ec
I20260619 06:34:09.667065 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:09.659746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:36201 (local address 127.28.104.194:51467)
0619 06:34:09.660899 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:09.660945 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:09.663153 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:09.663859 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:09.663870 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:09.663997 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:09.664741 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:34:09.664745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:09.664796 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:34:09.664916 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:09.665389 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:09.665402 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:09.665411 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:34:09.665666 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 06:34:09.665965 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":285,"threads_started":1}
I20260619 06:34:09.668681 1612 heartbeater.cc:347] Connected to a master server at 127.28.104.254:36201
I20260619 06:34:09.669097 1612 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:09.669816 1612 heartbeater.cc:511] Master 127.28.104.254:36201 requested a full tablet report, sending...
I20260619 06:34:09.669893 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36201
--builtin_ntp_servers=127.28.104.212:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 06:34:09.671306 1292 ts_manager.cc:194] Registered new tserver with Master: ed8a5f8308de422b9c62d7f97fd36dce (127.28.104.194:42613)
W20260619 06:34:10.000267 1615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:10.000542 1615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:10.000706 1615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:10.008072 1615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:10.008198 1615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:10.008363 1615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:34:10.017663 1615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:10.018960 1615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:10.020780 1615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:10.033710 1623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:10.033788 1620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:10.033746 1621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:10.038549 1615 server_base.cc:1034] running on GCE node
I20260619 06:34:10.039239 1615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:10.040220 1615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:10.041417 1615 hybrid_clock.cc:648] HybridClock initialized: now 1781850850041369 us; error 39 us; skew 500 ppm
I20260619 06:34:10.041738 1615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:10.044015 1615 webserver.cc:460] Webserver started at http://127.28.104.195:33497/ using document root <none> and password file <none>
I20260619 06:34:10.044718 1615 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:10.044831 1615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:10.045264 1615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:10.048195 1615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "b0344420c3d04afcb303c094986f34f9"
format_stamp: "Formatted at 2026-06-19 06:34:10 on dist-test-slave-nf4n"
server_key: "2468dbbf4bd7959a334a63cffe3c6c7d"
server_key_iv: "969e84199471401e2e2a8d7b6ef1e7e0"
server_key_version: "encryptionkey@0"
I20260619 06:34:10.048975 1615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "b0344420c3d04afcb303c094986f34f9"
format_stamp: "Formatted at 2026-06-19 06:34:10 on dist-test-slave-nf4n"
server_key: "2468dbbf4bd7959a334a63cffe3c6c7d"
server_key_iv: "969e84199471401e2e2a8d7b6ef1e7e0"
server_key_version: "encryptionkey@0"
I20260619 06:34:10.054109 1615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 06:34:10.057487 1630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:10.058771 1615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:34:10.058943 1615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "b0344420c3d04afcb303c094986f34f9"
format_stamp: "Formatted at 2026-06-19 06:34:10 on dist-test-slave-nf4n"
server_key: "2468dbbf4bd7959a334a63cffe3c6c7d"
server_key_iv: "969e84199471401e2e2a8d7b6ef1e7e0"
server_key_version: "encryptionkey@0"
I20260619 06:34:10.059165 1615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:10.064883 1615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:10.065680 1615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:10.065956 1615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:10.066833 1615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:10.068181 1615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:10.068297 1615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:10.068423 1615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:10.068506 1615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:10.100531 1615 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:33479
I20260619 06:34:10.100696 1742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:33479 every 8 connection(s)
I20260619 06:34:10.102043 1615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:34:10.110114 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.104529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:57669 (local address 127.28.104.254:36201)
0619 06:34:10.104770 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.104776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.106307 (+ 1531us) server_negotiation.cc:407] Connection header received
0619 06:34:10.108084 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.108088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.108143 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.108232 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.109539 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.109544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.109547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.109561 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.109587 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.109598 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.109696 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.109849 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.109899 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:34:10.111038 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 1615
I20260619 06:34:10.111198 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.105150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:36201 (local address 127.28.104.195:57669)
0619 06:34:10.106152 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.106197 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:10.107841 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:10.108445 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:10.108453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:10.108543 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:10.109260 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:34:10.109262 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:10.109300 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:34:10.109394 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:10.109677 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:10.109684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:10.109690 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:34:10.109899 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 06:34:10.110098 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":258,"threads_started":1}
I20260619 06:34:10.111230 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:34:10.111752 29091 external_mini_cluster.cc:1428] Setting key 0e42f19561fdbfb0196049e5d4164657
I20260619 06:34:10.112591 1743 heartbeater.cc:347] Connected to a master server at 127.28.104.254:36201
I20260619 06:34:10.112936 1743 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:10.113708 1743 heartbeater.cc:511] Master 127.28.104.254:36201 requested a full tablet report, sending...
I20260619 06:34:10.115163 1292 ts_manager.cc:194] Registered new tserver with Master: b0344420c3d04afcb303c094986f34f9 (127.28.104.195:33479)
I20260619 06:34:10.127398 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:34:10.134070 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.131713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.28.104.254:36201)
0619 06:34:10.131914 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.131922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.132365 (+ 443us) server_negotiation.cc:407] Connection header received
0619 06:34:10.132493 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.132496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.132548 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.132627 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.133478 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.133482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.133485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.133500 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.133531 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.133541 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.133629 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.133698 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.133742 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:34:10.137997 1292 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:34:10.145516 1292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:34:10.164814 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.158294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.28.104.193:36871)
0619 06:34:10.159447 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.159457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.159490 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:34:10.159793 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.159803 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.160054 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.160246 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.162807 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.162819 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.162839 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.162862 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.162928 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.162949 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.163178 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.164240 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.164370 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":805,"threads_started":1}
I20260619 06:34:10.165251 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.158139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36871 (local address 127.0.0.1:56304)
0619 06:34:10.158895 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.158940 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:10.159590 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:10.161003 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:10.161015 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:10.161148 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:10.162158 (+ 1010us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:34:10.162164 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:10.162215 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:34:10.162321 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:10.163257 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:10.163272 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:10.163284 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 06:34:10.163499 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 06:34:10.163611 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":353,"threads_started":1}
I20260619 06:34:10.169791 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.159280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:42613 (local address 127.0.0.1:50714)
0619 06:34:10.165531 (+ 6251us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.165555 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:10.165663 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:10.167739 (+ 2076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:10.167748 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:10.167807 (+ 59us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:10.168770 (+ 963us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:34:10.168773 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:10.168788 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:34:10.168833 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:10.169378 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:10.169383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:10.169387 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:34:10.169480 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:34:10.169529 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6121,"threads_started":1}
I20260619 06:34:10.170384 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.159338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.28.104.194:42613)
0619 06:34:10.159873 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.159885 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.165806 (+ 5921us) server_negotiation.cc:407] Connection header received
0619 06:34:10.165903 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.165916 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.166223 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.166394 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.169006 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.169016 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.169051 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.169071 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.169134 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.169171 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.169320 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.169979 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.170081 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":608,"threads_started":1}
I20260619 06:34:10.173736 1415 tablet_service.cc:1467] Processing CreateTablet for tablet a7b8d1149c85400b936af25f63e120be (DEFAULT_TABLE table=test-table [id=d37784329e7f45598c3a73a4201aef06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:10.174440 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.159830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:33479 (local address 127.0.0.1:48966)
0619 06:34:10.168479 (+ 8649us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.168497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:10.168582 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:10.172683 (+ 4101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:10.172688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:10.172724 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:10.173448 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:34:10.173451 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:10.173467 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:34:10.173507 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:10.173914 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:10.173919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:10.173923 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:34:10.174043 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 06:34:10.174087 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8517,"thread_start_us":8518,"threads_started":1}
I20260619 06:34:10.174942 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.160444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.28.104.195:33479)
0619 06:34:10.171172 (+ 10728us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.171180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.171216 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:34:10.171287 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.171295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.171494 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.171678 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.173678 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.173687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.173700 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.173719 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.173777 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.173798 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.173979 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.174042 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.174133 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10568,"thread_start_us":10398,"threads_started":1}
I20260619 06:34:10.175238 1415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b8d1149c85400b936af25f63e120be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:10.174762 1547 tablet_service.cc:1467] Processing CreateTablet for tablet a7b8d1149c85400b936af25f63e120be (DEFAULT_TABLE table=test-table [id=d37784329e7f45598c3a73a4201aef06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:10.176107 1547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b8d1149c85400b936af25f63e120be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:10.178166 1678 tablet_service.cc:1467] Processing CreateTablet for tablet a7b8d1149c85400b936af25f63e120be (DEFAULT_TABLE table=test-table [id=d37784329e7f45598c3a73a4201aef06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:10.179272 1678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b8d1149c85400b936af25f63e120be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:10.190092 1761 tablet_bootstrap.cc:492] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: Bootstrap starting.
I20260619 06:34:10.192472 1762 tablet_bootstrap.cc:492] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49: Bootstrap starting.
I20260619 06:34:10.194248 1763 tablet_bootstrap.cc:492] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: Bootstrap starting.
I20260619 06:34:10.194942 1761 tablet_bootstrap.cc:654] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:10.196018 1762 tablet_bootstrap.cc:654] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:10.196223 1761 log.cc:824] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:10.197151 1763 tablet_bootstrap.cc:654] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:10.197299 1762 log.cc:824] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:10.198400 1763 log.cc:824] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:10.200142 1761 tablet_bootstrap.cc:492] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: No bootstrap required, opened a new log
I20260619 06:34:10.200183 1762 tablet_bootstrap.cc:492] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49: No bootstrap required, opened a new log
I20260619 06:34:10.200441 1762 ts_tablet_manager.cc:1397] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260619 06:34:10.200426 1761 ts_tablet_manager.cc:1397] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 06:34:10.207412 1763 tablet_bootstrap.cc:492] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: No bootstrap required, opened a new log
I20260619 06:34:10.207700 1763 ts_tablet_manager.cc:1397] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260619 06:34:10.211133 1761 raft_consensus.cc:348] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 } } peers { permanent_uuid: "b0344420c3d04afcb303c094986f34f9" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 33479 } } peers { permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 } }
I20260619 06:34:10.211576 1761 raft_consensus.cc:374] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:10.211666 1761 raft_consensus.cc:729] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0344420c3d04afcb303c094986f34f9, State: Initialized, Role: FOLLOWER
I20260619 06:34:10.212146 1762 raft_consensus.cc:348] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 } } peers { permanent_uuid: "b0344420c3d04afcb303c094986f34f9" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 33479 } } peers { permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 } }
I20260619 06:34:10.212574 1762 raft_consensus.cc:374] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:10.212427 1761 consensus_queue.cc:260] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 } } peers { permanent_uuid: "b0344420c3d04afcb303c094986f34f9" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 33479 } } peers { permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 } }
I20260619 06:34:10.212669 1762 raft_consensus.cc:729] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aff50c176844836876e1f936f1edc49, State: Initialized, Role: FOLLOWER
I20260619 06:34:10.213466 1762 consensus_queue.cc:260] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 } } peers { permanent_uuid: "b0344420c3d04afcb303c094986f34f9" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 33479 } } peers { permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 } }
I20260619 06:34:10.218401 1763 raft_consensus.cc:348] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 } } peers { permanent_uuid: "b0344420c3d04afcb303c094986f34f9" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 33479 } } peers { permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 } }
I20260619 06:34:10.218793 1763 raft_consensus.cc:374] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:10.218914 1763 raft_consensus.cc:729] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8a5f8308de422b9c62d7f97fd36dce, State: Initialized, Role: FOLLOWER
I20260619 06:34:10.219583 1763 consensus_queue.cc:260] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 } } peers { permanent_uuid: "b0344420c3d04afcb303c094986f34f9" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 33479 } } peers { permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 } }
I20260619 06:34:10.219928 1481 heartbeater.cc:503] Master 127.28.104.254:36201 was elected leader, sending a full tablet report...
I20260619 06:34:10.221818 1612 heartbeater.cc:503] Master 127.28.104.254:36201 was elected leader, sending a full tablet report...
I20260619 06:34:10.222544 1763 ts_tablet_manager.cc:1428] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260619 06:34:10.223618 1761 ts_tablet_manager.cc:1428] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: Time spent starting tablet: real 0.023s user 0.013s sys 0.007s
I20260619 06:34:10.223973 1743 heartbeater.cc:503] Master 127.28.104.254:36201 was elected leader, sending a full tablet report...
I20260619 06:34:10.225277 1762 ts_tablet_manager.cc:1428] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49: Time spent starting tablet: real 0.025s user 0.011s sys 0.012s
W20260619 06:34:10.353749 1744 tablet.cc:2369] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:34:10.408736 1613 tablet.cc:2369] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:34:10.454766 1482 tablet.cc:2369] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:34:10.463157 1767 raft_consensus.cc:482] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:34:10.463482 1767 raft_consensus.cc:504] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 } } peers { permanent_uuid: "b0344420c3d04afcb303c094986f34f9" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 33479 } } peers { permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 } }
I20260619 06:34:10.465188 1767 leader_election.cc:288] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aff50c176844836876e1f936f1edc49 (127.28.104.193:36871), ed8a5f8308de422b9c62d7f97fd36dce (127.28.104.194:42613)
I20260619 06:34:10.467789 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.465800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:33911 (local address 127.28.104.193:36871)
0619 06:34:10.465979 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.465983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.466011 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:34:10.466061 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.466065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.466108 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.466209 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.467204 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.467209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.467212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.467226 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.467257 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.467268 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.467384 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.467496 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.467563 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:34:10.467821 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.465484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36871 (local address 127.28.104.195:33911)
0619 06:34:10.465677 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.465699 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:10.465807 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:10.466242 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:10.466248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:10.466279 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:10.466985 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:34:10.466988 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:10.467007 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:34:10.467055 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:10.467363 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:10.467370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:10.467375 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:34:10.467469 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:34:10.467511 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:34:10.468604 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.465643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:33181 (local address 127.28.104.194:42613)
0619 06:34:10.465831 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.465840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.466655 (+ 815us) server_negotiation.cc:407] Connection header received
0619 06:34:10.466753 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.466760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.466821 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.466918 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.467941 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.467946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.467952 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.467976 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.468021 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.468041 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.468161 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.468252 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.468311 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:34:10.468701 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.465484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:42613 (local address 127.28.104.195:33181)
0619 06:34:10.466557 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.466571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:10.466659 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:10.466978 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:10.466983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:10.467006 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:10.467718 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:34:10.467721 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:10.467735 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:34:10.467773 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:10.468133 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:10.468137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:10.468142 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:34:10.468262 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 06:34:10.468308 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":385,"thread_start_us":334,"threads_started":1}
I20260619 06:34:10.468869 1435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b8d1149c85400b936af25f63e120be" candidate_uuid: "b0344420c3d04afcb303c094986f34f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aff50c176844836876e1f936f1edc49" is_pre_election: true
I20260619 06:34:10.469499 1435 raft_consensus.cc:2454] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0344420c3d04afcb303c094986f34f9 in term 0.
I20260619 06:34:10.469681 1567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b8d1149c85400b936af25f63e120be" candidate_uuid: "b0344420c3d04afcb303c094986f34f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" is_pre_election: true
I20260619 06:34:10.470152 1567 raft_consensus.cc:2454] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0344420c3d04afcb303c094986f34f9 in term 0.
I20260619 06:34:10.470110 1632 leader_election.cc:302] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aff50c176844836876e1f936f1edc49, b0344420c3d04afcb303c094986f34f9; no voters:
I20260619 06:34:10.470541 1767 raft_consensus.cc:2781] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:34:10.470728 1767 raft_consensus.cc:482] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:34:10.470860 1767 raft_consensus.cc:3037] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:10.473074 1767 raft_consensus.cc:504] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 } } peers { permanent_uuid: "b0344420c3d04afcb303c094986f34f9" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 33479 } } peers { permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 } }
I20260619 06:34:10.473584 1767 leader_election.cc:288] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [CANDIDATE]: Term 1 election: Requested vote from peers 2aff50c176844836876e1f936f1edc49 (127.28.104.193:36871), ed8a5f8308de422b9c62d7f97fd36dce (127.28.104.194:42613)
I20260619 06:34:10.474027 1567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b8d1149c85400b936af25f63e120be" candidate_uuid: "b0344420c3d04afcb303c094986f34f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed8a5f8308de422b9c62d7f97fd36dce"
I20260619 06:34:10.474182 1567 raft_consensus.cc:3037] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:10.476214 1567 raft_consensus.cc:2454] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0344420c3d04afcb303c094986f34f9 in term 1.
I20260619 06:34:10.476544 1435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b8d1149c85400b936af25f63e120be" candidate_uuid: "b0344420c3d04afcb303c094986f34f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aff50c176844836876e1f936f1edc49"
I20260619 06:34:10.476606 1634 leader_election.cc:302] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0344420c3d04afcb303c094986f34f9, ed8a5f8308de422b9c62d7f97fd36dce; no voters:
I20260619 06:34:10.476814 1435 raft_consensus.cc:3037] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:10.476876 1767 raft_consensus.cc:2781] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:10.477406 1767 raft_consensus.cc:686] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 1 LEADER]: Becoming Leader. State: Replica: b0344420c3d04afcb303c094986f34f9, State: Running, Role: LEADER
I20260619 06:34:10.477954 1767 consensus_queue.cc:237] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 } } peers { permanent_uuid: "b0344420c3d04afcb303c094986f34f9" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 33479 } } peers { permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 } }
I20260619 06:34:10.479405 1435 raft_consensus.cc:2454] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0344420c3d04afcb303c094986f34f9 in term 1.
I20260619 06:34:10.482512 1291 catalog_manager.cc:5495] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0344420c3d04afcb303c094986f34f9 (127.28.104.195). New cstate: current_term: 1 leader_uuid: "b0344420c3d04afcb303c094986f34f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0344420c3d04afcb303c094986f34f9" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 33479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 } health_report { overall_health: UNKNOWN } } }
I20260619 06:34:10.551666 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.549608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.28.104.195:33479)
0619 06:34:10.549803 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.549809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.549828 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:34:10.549933 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.549937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.550034 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.550142 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.551063 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.551067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.551070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.551085 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.551116 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.551130 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.551253 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.551361 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.551421 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:34:10.560297 1567 raft_consensus.cc:1261] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce [term 1 FOLLOWER]: Refusing update from remote peer b0344420c3d04afcb303c094986f34f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:10.560300 1435 raft_consensus.cc:1261] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49 [term 1 FOLLOWER]: Refusing update from remote peer b0344420c3d04afcb303c094986f34f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:10.561339 1771 consensus_queue.cc:1035] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed8a5f8308de422b9c62d7f97fd36dce" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:10.561755 1767 consensus_queue.cc:1035] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aff50c176844836876e1f936f1edc49" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:10.574076 1775 mvcc.cc:205] Tried to move back new op lower bound from 7298461083885199360 to 7298461083561422848. Current Snapshot: MvccSnapshot[applied={T|T < 7298461083885199360}]
I20260619 06:34:10.577245 1777 mvcc.cc:205] Tried to move back new op lower bound from 7298461083885199360 to 7298461083561422848. Current Snapshot: MvccSnapshot[applied={T|T < 7298461083885199360}]
I20260619 06:34:10.585624 1292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:10.585878 1292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:10.589313 1292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af80a1eede3045b2bb4bdf2ec9b07ba3: Sending DeleteTablet for 3 replicas of tablet a7b8d1149c85400b936af25f63e120be
I20260619 06:34:10.590477 1547 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b8d1149c85400b936af25f63e120be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:10 UTC) from {username='slave'} at 127.0.0.1:50714
I20260619 06:34:10.590595 1415 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b8d1149c85400b936af25f63e120be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:10 UTC) from {username='slave'} at 127.0.0.1:56304
I20260619 06:34:10.591382 1784 tablet_replica.cc:330] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: stopping tablet replica
I20260619 06:34:10.591753 1678 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b8d1149c85400b936af25f63e120be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:10 UTC) from {username='slave'} at 127.0.0.1:48966
I20260619 06:34:10.591816 1784 raft_consensus.cc:2229] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:10.592312 1784 raft_consensus.cc:2258] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:10.592391 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 1352
I20260619 06:34:10.602226 1784 ts_tablet_manager.cc:1905] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:10.604372 1786 tablet_replica.cc:330] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: stopping tablet replica
I20260619 06:34:10.605497 1786 raft_consensus.cc:2229] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:34:10.606024 1785 tablet_replica.cc:330] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49: stopping tablet replica
I20260619 06:34:10.606498 1786 raft_consensus.cc:2258] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:10.606540 1785 raft_consensus.cc:2229] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:10.607178 1785 raft_consensus.cc:2258] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:10.609262 1786 ts_tablet_manager.cc:1905] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:10.609390 1785 ts_tablet_manager.cc:1905] T a7b8d1149c85400b936af25f63e120be P 2aff50c176844836876e1f936f1edc49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:10.610162 1784 ts_tablet_manager.cc:1918] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:10.610335 1784 log.cc:1196] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/a7b8d1149c85400b936af25f63e120be
I20260619 06:34:10.612879 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.594369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.28.104.193:36871)
0619 06:34:10.594570 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.594577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.609988 (+ 15411us) server_negotiation.cc:407] Connection header received
0619 06:34:10.610608 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.610615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.610714 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.610832 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.612083 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.612089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.612098 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.612118 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.612155 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.612169 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.612270 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.612509 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.612578 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:34:10.613950 1476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:10.617566 1786 ts_tablet_manager.cc:1918] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:10.617725 1786 log.cc:1196] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/a7b8d1149c85400b936af25f63e120be
I20260619 06:34:10.618295 1786 ts_tablet_manager.cc:1939] T a7b8d1149c85400b936af25f63e120be P b0344420c3d04afcb303c094986f34f9: Deleting consensus metadata
I20260619 06:34:10.620424 1276 catalog_manager.cc:4841] TS b0344420c3d04afcb303c094986f34f9 (127.28.104.195:33479): tablet a7b8d1149c85400b936af25f63e120be (table test-table [id=d37784329e7f45598c3a73a4201aef06]) successfully deleted
I20260619 06:34:10.621667 1784 ts_tablet_manager.cc:1939] T a7b8d1149c85400b936af25f63e120be P ed8a5f8308de422b9c62d7f97fd36dce: Deleting consensus metadata
I20260619 06:34:10.623675 1278 catalog_manager.cc:4841] TS ed8a5f8308de422b9c62d7f97fd36dce (127.28.104.194:42613): tablet a7b8d1149c85400b936af25f63e120be (table test-table [id=d37784329e7f45598c3a73a4201aef06]) successfully deleted
I20260619 06:34:10.722086 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 1352
W20260619 06:34:10.740049 1276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.104.193:36871 (error 108)
I20260619 06:34:10.740238 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 1485
I20260619 06:34:10.742733 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.741541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36871 (local address 127.0.0.1:56334)
0619 06:34:10.742392 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.742525 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":590,"threads_started":1}
W20260619 06:34:10.743124 1276 catalog_manager.cc:4584] TS 2aff50c176844836876e1f936f1edc49 (127.28.104.193:36871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7b8d1149c85400b936af25f63e120be: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36871: connect: Connection refused (error 111)
I20260619 06:34:10.743422 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.741237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.28.104.194:42613)
0619 06:34:10.741413 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.741419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.741442 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:34:10.741509 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.741515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.741610 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.741721 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.742735 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.742740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.742747 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.742769 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.742807 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.742825 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.742960 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.743065 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.743126 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:34:10.744024 1607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:10.796752 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.796253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36871 (local address 127.0.0.1:56340)
0619 06:34:10.796479 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.796592 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 06:34:10.797962 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.797573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36871 (local address 127.0.0.1:56346)
0619 06:34:10.797724 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.797816 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 06:34:10.858563 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.858120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36871 (local address 127.0.0.1:56362)
0619 06:34:10.858320 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.858422 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 06:34:10.859637 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.859271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36871 (local address 127.0.0.1:56366)
0619 06:34:10.859429 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.859506 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 06:34:10.892655 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 1485
I20260619 06:34:10.910291 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 1615
I20260619 06:34:10.913564 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.911181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.28.104.195:33479)
0619 06:34:10.911375 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:10.911379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:10.911398 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:34:10.911627 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:10.911631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:10.911683 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:10.911792 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:10.912722 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:10.912726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:10.912729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:10.912772 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:10.912805 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:10.912817 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:10.912936 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:10.913166 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 06:34:10.913239 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 06:34:10.914297 1738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:10.939896 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.939354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36871 (local address 127.0.0.1:56380)
0619 06:34:10.939564 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.939708 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 06:34:10.941078 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:10.940559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36871 (local address 127.0.0.1:56388)
0619 06:34:10.940802 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:10.940891 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 06:34:11.034749 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 1615
I20260619 06:34:11.053644 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 1260
I20260619 06:34:11.054862 1323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:11.160910 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:11.160175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36871 (local address 127.0.0.1:56394)
0619 06:34:11.160564 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:11.160704 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260619 06:34:11.160888 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 1260
2026-06-19T06:34:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2966 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 06:34:11.198513 29091 mini_oidc.cc:152] Starting JWKS server
I20260619 06:34:11.200984 29091 webserver.cc:460] Webserver started at https://127.0.0.1:37377/ using document root <none> and password file <none>
I20260619 06:34:11.201339 29091 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:34:11.201927 29091 webserver.cc:460] Webserver started at http://127.0.0.1:45347/ using document root <none> and password file <none>
2026-06-19T06:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:11Z Disabled control of system clock
I20260619 06:34:11.230021 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:45451
--webserver_interface=127.28.104.254
--webserver_port=0
--builtin_ntp_servers=127.28.104.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:45451
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37377/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/testchainca.cert with env {}
W20260619 06:34:11.560428 1799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:11.560776 1799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:11.560935 1799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:11.568171 1799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:11.568344 1799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:11.568434 1799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:11.568549 1799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:11.568630 1799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:34:11.568720 1799 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:34:11.568795 1799 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37377/jwks.json
I20260619 06:34:11.578934 1799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:45451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:45451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37377/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:11.580335 1799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:11.581727 1799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:11.593523 1804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:11.593556 1805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:11.594187 1807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:11.594684 1799 server_base.cc:1034] running on GCE node
I20260619 06:34:11.595499 1799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:11.597044 1799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:11.598343 1799 hybrid_clock.cc:648] HybridClock initialized: now 1781850851598280 us; error 48 us; skew 500 ppm
I20260619 06:34:11.598711 1799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:11.601114 1799 webserver.cc:460] Webserver started at http://127.28.104.254:40231/ using document root <none> and password file <none>
I20260619 06:34:11.601801 1799 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:11.601930 1799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:11.602274 1799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:11.605293 1799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e"
format_stamp: "Formatted at 2026-06-19 06:34:11 on dist-test-slave-nf4n"
server_key: "5b83f01a82a3d79272e9aaa83592f120"
server_key_iv: "b3dde3a31a4a892722123678e32b6300"
server_key_version: "encryptionkey@0"
I20260619 06:34:11.606117 1799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e"
format_stamp: "Formatted at 2026-06-19 06:34:11 on dist-test-slave-nf4n"
server_key: "5b83f01a82a3d79272e9aaa83592f120"
server_key_iv: "b3dde3a31a4a892722123678e32b6300"
server_key_version: "encryptionkey@0"
I20260619 06:34:11.611301 1799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:34:11.614744 1813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:11.616083 1799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:34:11.616328 1799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e"
format_stamp: "Formatted at 2026-06-19 06:34:11 on dist-test-slave-nf4n"
server_key: "5b83f01a82a3d79272e9aaa83592f120"
server_key_iv: "b3dde3a31a4a892722123678e32b6300"
server_key_version: "encryptionkey@0"
I20260619 06:34:11.616577 1799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:11.630990 1799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:11.631914 1799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:11.632275 1799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:11.651854 1799 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:45451
I20260619 06:34:11.651886 1864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:45451 every 8 connection(s)
I20260619 06:34:11.653394 1799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:11.657140 1865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:11.658557 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 1799
I20260619 06:34:11.658753 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:11.659543 29091 external_mini_cluster.cc:1428] Setting key 71a9da30a889fdb858c380821fb8db0a
I20260619 06:34:11.670226 1865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e: Bootstrap starting.
I20260619 06:34:11.673519 1865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:11.673777 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:11.661606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.28.104.254:45451)
0619 06:34:11.662650 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:11.662660 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:11.662704 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:34:11.664393 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:11.664416 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:11.664890 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:11.665417 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:11.666013 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:11.667340 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:11.668779 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:11.669539 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:11.669785 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:11.671152 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:11.671161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:11.671175 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:11.671790 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:11.671886 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:11.671908 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:11.672056 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:11.672306 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 06:34:11.672623 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":351,"threads_started":1}
I20260619 06:34:11.674909 1865 log.cc:824] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:11.677896 1865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e: No bootstrap required, opened a new log
I20260619 06:34:11.677937 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:34:11.685386 1865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45451 } }
I20260619 06:34:11.685832 1865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:11.685979 1865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efe2705ff8d48d4b5f17b7c34d7bd2e, State: Initialized, Role: FOLLOWER
I20260619 06:34:11.686672 1865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45451 } }
I20260619 06:34:11.686916 1865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:11.687050 1865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:11.687216 1865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:11.689443 1865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45451 } }
I20260619 06:34:11.689937 1865 leader_election.cc:302] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8efe2705ff8d48d4b5f17b7c34d7bd2e; no voters:
I20260619 06:34:11.690495 1865 leader_election.cc:288] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:11.690650 1870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:11.691334 1870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [term 1 LEADER]: Becoming Leader. State: Replica: 8efe2705ff8d48d4b5f17b7c34d7bd2e, State: Running, Role: LEADER
I20260619 06:34:11.691901 1870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45451 } }
I20260619 06:34:11.692672 1865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:11.694805 1872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8efe2705ff8d48d4b5f17b7c34d7bd2e. Latest consensus state: current_term: 1 leader_uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45451 } } }
I20260619 06:34:11.695219 1872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:11.696008 1871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efe2705ff8d48d4b5f17b7c34d7bd2e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45451 } } }
I20260619 06:34:11.696334 1871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:11.696903 1875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:11.703683 1875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:11.714185 1875 catalog_manager.cc:1347] Generated new cluster ID: 98f97889a0d44f218a641e787312b2b9
I20260619 06:34:11.714341 1875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:11.729476 1875 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:11.731755 1875 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:11.740479 1875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8efe2705ff8d48d4b5f17b7c34d7bd2e: Generated new TSK 0
I20260619 06:34:11.741415 1875 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:34:11.781994 1895 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 06:34:11.782172 1895 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:34:11.783396 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:11.761748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.28.104.254:45451)
0619 06:34:11.761937 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:11.761944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:11.762518 (+ 574us) server_negotiation.cc:407] Connection header received
0619 06:34:11.762793 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:11.762801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:11.762902 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:11.763064 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:34:11.763553 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:11.764686 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:11.766176 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:11.766871 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:11.766997 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:11.767486 (+ 489us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:34:11.782683 (+ 15197us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:34:11.783071 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51846: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 06:34:11.783520 1868 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51846: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 06:34:11.784811 1894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:11.761734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45451 (local address 127.0.0.1:51846)
0619 06:34:11.762421 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:11.762437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:11.762583 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:11.763085 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:11.763088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:11.763098 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:34:11.763409 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:11.763416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:11.764840 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:11.764843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:11.766037 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:11.766049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:11.767031 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:11.767036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:11.767155 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:11.767209 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:34:11.783045 (+ 15836us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:34:11.783382 (+ 337us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.104.254:45451: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":353,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.104.254:45451: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 06:34:11.786258 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 1799
I20260619 06:34:11.787600 1860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:11.882376 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 1799
2026-06-19T06:34:11Z chronyd exiting
I20260619 06:34:11.932265 29091 test_util.cc:175] -----------------------------------------------
I20260619 06:34:11.932385 29091 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850805520649-29091-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (738 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 06:34:11.936076 29091 mini_oidc.cc:152] Starting JWKS server
I20260619 06:34:11.937906 29091 webserver.cc:460] Webserver started at https://127.0.0.1:46203/ using document root <none> and password file <none>
I20260619 06:34:11.938105 29091 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:34:11.938692 29091 webserver.cc:460] Webserver started at http://127.0.0.1:36827/ using document root <none> and password file <none>
2026-06-19T06:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:11Z Disabled control of system clock
I20260619 06:34:11.966421 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:44443
--webserver_interface=127.28.104.254
--webserver_port=0
--builtin_ntp_servers=127.28.104.212:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:44443
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46203/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/testchainca.cert with env {}
W20260619 06:34:12.299302 1902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:12.299695 1902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:12.299875 1902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:12.307397 1902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:12.307574 1902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:12.307675 1902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:12.307788 1902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:12.307878 1902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:34:12.307979 1902 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:34:12.308060 1902 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46203/jwks.json
I20260619 06:34:12.318492 1902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:44443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:44443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46203/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:12.319970 1902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:12.321442 1902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:12.334196 1907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:12.334159 1910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:12.334111 1908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:12.334978 1902 server_base.cc:1034] running on GCE node
I20260619 06:34:12.335759 1902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:12.337380 1902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:12.338634 1902 hybrid_clock.cc:648] HybridClock initialized: now 1781850852338558 us; error 67 us; skew 500 ppm
I20260619 06:34:12.339048 1902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:12.341542 1902 webserver.cc:460] Webserver started at http://127.28.104.254:39719/ using document root <none> and password file <none>
I20260619 06:34:12.342233 1902 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:12.342365 1902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:12.342702 1902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:12.345737 1902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "a5b24e3e9cb7466fb46777ca82e55a6e"
format_stamp: "Formatted at 2026-06-19 06:34:12 on dist-test-slave-nf4n"
server_key: "284ea1064a539eaf641eb1174981c4e8"
server_key_iv: "a0b02ac60e74113e2a7ddb07346a7d8c"
server_key_version: "encryptionkey@0"
I20260619 06:34:12.346516 1902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "a5b24e3e9cb7466fb46777ca82e55a6e"
format_stamp: "Formatted at 2026-06-19 06:34:12 on dist-test-slave-nf4n"
server_key: "284ea1064a539eaf641eb1174981c4e8"
server_key_iv: "a0b02ac60e74113e2a7ddb07346a7d8c"
server_key_version: "encryptionkey@0"
I20260619 06:34:12.351804 1902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:34:12.355304 1916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:12.356657 1902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 06:34:12.356850 1902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "a5b24e3e9cb7466fb46777ca82e55a6e"
format_stamp: "Formatted at 2026-06-19 06:34:12 on dist-test-slave-nf4n"
server_key: "284ea1064a539eaf641eb1174981c4e8"
server_key_iv: "a0b02ac60e74113e2a7ddb07346a7d8c"
server_key_version: "encryptionkey@0"
I20260619 06:34:12.357087 1902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:12.378348 1902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:12.379205 1902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:12.379521 1902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:12.397598 1902 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:44443
I20260619 06:34:12.397617 1968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:44443 every 8 connection(s)
I20260619 06:34:12.398985 1902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:12.402920 1969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:12.405570 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 1902
I20260619 06:34:12.405800 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:12.406474 29091 external_mini_cluster.cc:1428] Setting key 02648b2c6079b4854e349b3d63abeec2
I20260619 06:34:12.415419 1969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e: Bootstrap starting.
I20260619 06:34:12.419255 1969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:12.420382 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:12.408632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.28.104.254:44443)
0619 06:34:12.409735 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:12.409746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:12.409788 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:34:12.411413 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:12.411435 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:12.411927 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:12.412264 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:12.412903 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:12.414246 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:12.416015 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:12.416656 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:12.416788 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:12.418089 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:12.418096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:12.418109 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:12.418649 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:12.418742 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:12.418763 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:12.418865 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:12.419090 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 06:34:12.419416 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":444,"threads_started":1}
I20260619 06:34:12.420389 1969 log.cc:824] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:12.423931 1969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e: No bootstrap required, opened a new log
I20260619 06:34:12.424676 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:34:12.431775 1969 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b24e3e9cb7466fb46777ca82e55a6e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44443 } }
I20260619 06:34:12.432153 1969 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:12.432262 1969 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b24e3e9cb7466fb46777ca82e55a6e, State: Initialized, Role: FOLLOWER
I20260619 06:34:12.432803 1969 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b24e3e9cb7466fb46777ca82e55a6e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44443 } }
I20260619 06:34:12.433084 1969 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:12.433225 1969 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:12.433398 1969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:12.435572 1969 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b24e3e9cb7466fb46777ca82e55a6e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44443 } }
I20260619 06:34:12.436048 1969 leader_election.cc:302] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5b24e3e9cb7466fb46777ca82e55a6e; no voters:
I20260619 06:34:12.436910 1969 leader_election.cc:288] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:12.437099 1974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:12.437801 1974 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [term 1 LEADER]: Becoming Leader. State: Replica: a5b24e3e9cb7466fb46777ca82e55a6e, State: Running, Role: LEADER
I20260619 06:34:12.438351 1974 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b24e3e9cb7466fb46777ca82e55a6e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44443 } }
I20260619 06:34:12.439008 1969 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:12.441320 1975 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5b24e3e9cb7466fb46777ca82e55a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b24e3e9cb7466fb46777ca82e55a6e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44443 } } }
I20260619 06:34:12.441841 1975 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:12.441455 1976 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5b24e3e9cb7466fb46777ca82e55a6e. Latest consensus state: current_term: 1 leader_uuid: "a5b24e3e9cb7466fb46777ca82e55a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b24e3e9cb7466fb46777ca82e55a6e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44443 } } }
I20260619 06:34:12.442286 1976 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:12.443657 1983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:12.448302 1983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:12.456753 1983 catalog_manager.cc:1347] Generated new cluster ID: 994afe1c912a463dbc7ba85d09cd5788
I20260619 06:34:12.456872 1983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:12.463995 1983 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:12.465806 1983 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:12.480160 1983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5b24e3e9cb7466fb46777ca82e55a6e: Generated new TSK 0
I20260619 06:34:12.481328 1983 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:34:12.524791 1999 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 06:34:12.524928 1999 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:34:12.526289 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:12.506037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.28.104.254:44443)
0619 06:34:12.506242 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:12.506250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:12.506738 (+ 488us) server_negotiation.cc:407] Connection header received
0619 06:34:12.506979 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:12.506985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:12.507076 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:12.507259 (+ 183us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:34:12.507653 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:12.508798 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:12.510171 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:12.510849 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:12.510997 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:12.511433 (+ 436us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:34:12.525542 (+ 14109us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46203/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:34:12.525961 (+ 419us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46203/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 06:34:12.526422 1972 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46203/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 06:34:12.526422 1998 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:12.505990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:44443 (local address 127.0.0.1:48804)
0619 06:34:12.506635 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:12.506655 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:12.506776 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:12.507263 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:12.507266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:12.507275 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:34:12.507503 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:12.507511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:12.508930 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:12.508933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:12.510038 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:12.510049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:12.511006 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:12.511010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:12.511140 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:12.511171 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:34:12.525842 (+ 14671us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46203/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:34:12.526067 (+ 225us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.104.254:44443: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46203/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":367,"threads_started":1}
I20260619 06:34:12.528131 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 1902
I20260619 06:34:12.529363 1964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:12.622058 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 1902
2026-06-19T06:34:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (741 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 06:34:12.678273 29091 mini_oidc.cc:152] Starting JWKS server
I20260619 06:34:12.680341 29091 webserver.cc:460] Webserver started at https://127.0.0.1:39711/ using document root <none> and password file <none>
I20260619 06:34:12.680600 29091 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:34:12.681303 29091 webserver.cc:460] Webserver started at http://127.0.0.1:36667/ using document root <none> and password file <none>
2026-06-19T06:34:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:12Z Disabled control of system clock
I20260619 06:34:12.708884 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:43817
--webserver_interface=127.28.104.254
--webserver_port=0
--builtin_ntp_servers=127.28.104.212:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:43817
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39711/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:34:13.034778 2006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:13.035105 2006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:13.035192 2006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:13.042472 2006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:13.042586 2006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:13.042624 2006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:13.042680 2006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:13.042725 2006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:34:13.042779 2006 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:34:13.042819 2006 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39711/jwks.json
I20260619 06:34:13.052196 2006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:43817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:43817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39711/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:13.053565 2006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:13.055330 2006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:13.067452 2011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:13.067435 2012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:13.067435 2014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:13.068572 2006 server_base.cc:1034] running on GCE node
I20260619 06:34:13.069365 2006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:13.070763 2006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:13.071956 2006 hybrid_clock.cc:648] HybridClock initialized: now 1781850853071915 us; error 35 us; skew 500 ppm
I20260619 06:34:13.072300 2006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:13.074509 2006 webserver.cc:460] Webserver started at http://127.28.104.254:42229/ using document root <none> and password file <none>
I20260619 06:34:13.075186 2006 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:13.075297 2006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:13.075639 2006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:13.078527 2006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "375366500096499faf0d0f012b175ce5"
format_stamp: "Formatted at 2026-06-19 06:34:13 on dist-test-slave-nf4n"
server_key: "b1b97831a2a59ce35e4a2d6ffebb1a21"
server_key_iv: "46c5e748824192bd2f399fea62a30c42"
server_key_version: "encryptionkey@0"
I20260619 06:34:13.079280 2006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "375366500096499faf0d0f012b175ce5"
format_stamp: "Formatted at 2026-06-19 06:34:13 on dist-test-slave-nf4n"
server_key: "b1b97831a2a59ce35e4a2d6ffebb1a21"
server_key_iv: "46c5e748824192bd2f399fea62a30c42"
server_key_version: "encryptionkey@0"
I20260619 06:34:13.084484 2006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:34:13.088274 2020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:13.089834 2006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:13.090032 2006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "375366500096499faf0d0f012b175ce5"
format_stamp: "Formatted at 2026-06-19 06:34:13 on dist-test-slave-nf4n"
server_key: "b1b97831a2a59ce35e4a2d6ffebb1a21"
server_key_iv: "46c5e748824192bd2f399fea62a30c42"
server_key_version: "encryptionkey@0"
I20260619 06:34:13.090235 2006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:13.105304 2006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:13.106134 2006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:13.106442 2006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:13.124388 2006 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:43817
I20260619 06:34:13.124406 2071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:43817 every 8 connection(s)
I20260619 06:34:13.125841 2006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:13.127818 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 2006
I20260619 06:34:13.128021 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:13.128839 29091 external_mini_cluster.cc:1428] Setting key 9b93521b888fb6c974600745d491300b
I20260619 06:34:13.130613 2072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:13.141733 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:13.130827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.28.104.254:43817)
0619 06:34:13.131799 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:13.131809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:13.131850 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:34:13.133441 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:13.133463 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:13.133922 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:13.134314 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:13.134862 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.136158 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.137501 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.138145 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.138362 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:13.139501 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:13.139508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:13.139521 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:13.140073 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:13.140148 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:13.140175 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:13.140297 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:13.140499 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 06:34:13.140789 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":339,"threads_started":1}
I20260619 06:34:13.141885 2072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5: Bootstrap starting.
I20260619 06:34:13.145138 2072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:13.145309 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:34:13.146178 2072 log.cc:824] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:13.149158 2072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5: No bootstrap required, opened a new log
I20260619 06:34:13.156394 2072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375366500096499faf0d0f012b175ce5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43817 } }
I20260619 06:34:13.156764 2072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:13.156895 2072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375366500096499faf0d0f012b175ce5, State: Initialized, Role: FOLLOWER
I20260619 06:34:13.157473 2072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375366500096499faf0d0f012b175ce5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43817 } }
I20260619 06:34:13.157704 2072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:13.157856 2072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:13.158017 2072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:13.160112 2072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375366500096499faf0d0f012b175ce5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43817 } }
I20260619 06:34:13.160595 2072 leader_election.cc:302] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 375366500096499faf0d0f012b175ce5; no voters:
I20260619 06:34:13.161280 2072 leader_election.cc:288] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:13.161417 2077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:13.162146 2077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 375366500096499faf0d0f012b175ce5, State: Running, Role: LEADER
I20260619 06:34:13.162719 2077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375366500096499faf0d0f012b175ce5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43817 } }
I20260619 06:34:13.163408 2072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:13.165786 2079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 375366500096499faf0d0f012b175ce5. Latest consensus state: current_term: 1 leader_uuid: "375366500096499faf0d0f012b175ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375366500096499faf0d0f012b175ce5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43817 } } }
I20260619 06:34:13.165741 2078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "375366500096499faf0d0f012b175ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375366500096499faf0d0f012b175ce5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43817 } } }
I20260619 06:34:13.166152 2079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:13.166220 2078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:13.167806 2085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:13.173210 2085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:13.182585 2085 catalog_manager.cc:1347] Generated new cluster ID: 3964934ad53c4487803c3c655d22357b
I20260619 06:34:13.182704 2085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:13.191665 2085 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:13.193466 2085 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:13.207175 2085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 375366500096499faf0d0f012b175ce5: Generated new TSK 0
I20260619 06:34:13.208276 2085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:34:13.241065 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:13.232812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.28.104.254:43817)
0619 06:34:13.233109 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:13.233116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:13.233499 (+ 383us) server_negotiation.cc:407] Connection header received
0619 06:34:13.233729 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:13.233736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:13.233812 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:13.233948 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:34:13.234510 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.235239 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.237304 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.237744 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.237847 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:13.238445 (+ 598us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:34:13.240379 (+ 1934us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39711/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:34:13.240722 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39711/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 06:34:13.241322 2101 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:13.232711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:43817 (local address 127.0.0.1:52344)
0619 06:34:13.233381 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:13.233404 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:13.233559 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:13.233987 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:13.233993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:13.234006 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:34:13.234351 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:13.234363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.235421 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.235427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:13.237163 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:13.237179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.237927 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.237933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:13.238119 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:13.238164 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:34:13.240581 (+ 2417us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39711/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:34:13.240856 (+ 275us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.104.254:43817: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39711/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":376,"threads_started":1}
W20260619 06:34:13.241484 2075 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39711/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260619 06:34:13.242499 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 2006
I20260619 06:34:13.243657 2067 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 06:34:13.258965 2102 webserver.cc:533] Webserver: sslize failed:
E20260619 06:34:13.259089 2102 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:34:13.340137 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 2006
2026-06-19T06:34:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (719 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T06:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:13Z Disabled control of system clock
I20260619 06:34:13.423779 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:38949
--webserver_interface=127.28.104.254
--webserver_port=0
--builtin_ntp_servers=127.28.104.212:39593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:38949
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/testchainca.cert with env {}
W20260619 06:34:13.751578 2107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:13.751912 2107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:13.752007 2107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:13.760730 2107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:13.760864 2107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:13.760903 2107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:13.760954 2107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:13.760998 2107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:34:13.761125 2107 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:34:13.761175 2107 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 06:34:13.772178 2107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:38949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:38949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:13.773703 2107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:13.775068 2107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:13.787395 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:13.787513 2115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:13.787603 2113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:13.788964 2107 server_base.cc:1034] running on GCE node
I20260619 06:34:13.789805 2107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:13.791216 2107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:13.792412 2107 hybrid_clock.cc:648] HybridClock initialized: now 1781850853792371 us; error 30 us; skew 500 ppm
I20260619 06:34:13.792765 2107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:13.795040 2107 webserver.cc:460] Webserver started at http://127.28.104.254:40939/ using document root <none> and password file <none>
I20260619 06:34:13.795719 2107 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:13.795830 2107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:13.796140 2107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:13.798987 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "99380638eb1140c38512fd1198bb6505"
format_stamp: "Formatted at 2026-06-19 06:34:13 on dist-test-slave-nf4n"
server_key: "562b96748862fb04723257d6123e9200"
server_key_iv: "65758c90697ba9e1f6a243c8c7ca2552"
server_key_version: "encryptionkey@0"
I20260619 06:34:13.799767 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "99380638eb1140c38512fd1198bb6505"
format_stamp: "Formatted at 2026-06-19 06:34:13 on dist-test-slave-nf4n"
server_key: "562b96748862fb04723257d6123e9200"
server_key_iv: "65758c90697ba9e1f6a243c8c7ca2552"
server_key_version: "encryptionkey@0"
I20260619 06:34:13.804862 2107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 06:34:13.808614 2121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:13.810325 2107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:13.810549 2107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "99380638eb1140c38512fd1198bb6505"
format_stamp: "Formatted at 2026-06-19 06:34:13 on dist-test-slave-nf4n"
server_key: "562b96748862fb04723257d6123e9200"
server_key_iv: "65758c90697ba9e1f6a243c8c7ca2552"
server_key_version: "encryptionkey@0"
I20260619 06:34:13.810786 2107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:13.853971 2107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:13.854928 2107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:13.855257 2107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:13.874238 2107 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:38949
I20260619 06:34:13.874253 2174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:38949 every 8 connection(s)
I20260619 06:34:13.875689 2107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:13.879549 2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:13.884013 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 2107
I20260619 06:34:13.884249 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:13.885082 29091 external_mini_cluster.cc:1428] Setting key 7c01bc5ea248d12e58187dfc3814b82a
I20260619 06:34:13.891446 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505: Bootstrap starting.
I20260619 06:34:13.894687 2175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:13.895733 2175 log.cc:824] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:13.898679 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505: No bootstrap required, opened a new log
I20260619 06:34:13.898904 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:13.887078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.28.104.254:38949)
0619 06:34:13.888199 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:13.888223 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:13.888265 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:34:13.890116 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:13.890142 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:13.890645 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:13.891055 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:13.891625 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.892944 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.894281 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.895046 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.895255 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:13.896459 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:13.896466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:13.896480 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:13.897169 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:13.897258 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:13.897280 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:13.897395 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:13.897636 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 06:34:13.897945 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":388,"threads_started":1}
I20260619 06:34:13.902890 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:34:13.907122 2175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99380638eb1140c38512fd1198bb6505" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38949 } }
I20260619 06:34:13.907522 2175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:13.907636 2175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99380638eb1140c38512fd1198bb6505, State: Initialized, Role: FOLLOWER
I20260619 06:34:13.908227 2175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99380638eb1140c38512fd1198bb6505" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38949 } }
I20260619 06:34:13.908629 2175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:13.908787 2175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:13.908982 2175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:13.911536 2175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99380638eb1140c38512fd1198bb6505" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38949 } }
I20260619 06:34:13.912062 2175 leader_election.cc:302] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99380638eb1140c38512fd1198bb6505; no voters:
I20260619 06:34:13.912674 2175 leader_election.cc:288] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:13.912823 2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:13.913571 2180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [term 1 LEADER]: Becoming Leader. State: Replica: 99380638eb1140c38512fd1198bb6505, State: Running, Role: LEADER
I20260619 06:34:13.914189 2180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99380638eb1140c38512fd1198bb6505" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38949 } }
I20260619 06:34:13.915053 2175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:13.917303 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99380638eb1140c38512fd1198bb6505. Latest consensus state: current_term: 1 leader_uuid: "99380638eb1140c38512fd1198bb6505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99380638eb1140c38512fd1198bb6505" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38949 } } }
I20260619 06:34:13.917250 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99380638eb1140c38512fd1198bb6505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99380638eb1140c38512fd1198bb6505" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38949 } } }
I20260619 06:34:13.917651 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:13.917651 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:13.919817 2187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:13.925244 2187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:13.934311 2187 catalog_manager.cc:1347] Generated new cluster ID: 684fa42d237347dc9bd6d006393e36eb
I20260619 06:34:13.934450 2187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:13.960610 2187 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:13.962126 2187 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:13.972034 2187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99380638eb1140c38512fd1198bb6505: Generated new TSK 0
I20260619 06:34:13.972916 2187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:34:14.446172 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:13.985185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.28.104.254:38949)
0619 06:34:13.985386 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:13.985393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:13.985989 (+ 596us) server_negotiation.cc:407] Connection header received
0619 06:34:13.986263 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:13.986269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:13.986346 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:13.986475 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:34:13.986976 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.987687 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.989847 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.990302 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.990402 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:13.990957 (+ 555us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:34:14.445589 (+454632us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:34:14.445938 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45812: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 06:34:14.446283 2178 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45812: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 06:34:14.446480 2204 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:13.985173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:38949 (local address 127.0.0.1:45812)
0619 06:34:13.985873 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:13.985896 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:13.986060 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:13.986517 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:13.986522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:13.986535 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:34:13.986832 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:13.986844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.987927 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.987933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:13.989705 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:13.989724 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:13.990472 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:13.990479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:13.990654 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:13.990700 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:34:14.445873 (+455173us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:34:14.446115 (+ 242us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.104.254:38949: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":343,"threads_started":1}
I20260619 06:34:14.447636 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 2107
I20260619 06:34:14.448805 2170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:14.551397 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 2107
2026-06-19T06:34:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1191 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (424 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (420 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:34:15 dist-test-slave-nf4n krb5kdc[2219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:34:15 dist-test-slave-nf4n krb5kdc[2219](info): set up 2 sockets
Jun 19 06:34:15 dist-test-slave-nf4n krb5kdc[2219](info): commencing operation
krb5kdc: starting...
W20260619 06:34:16.510085 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:34:16 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:34:16.723912 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:34677
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:34677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:17.053300 2233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:17.053584 2233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:17.053759 2233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:17.060957 2233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:17.061087 2233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:17.061210 2233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:17.061316 2233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:17.061406 2233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:17.071190 2233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:34677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:34677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:17.072623 2233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:17.074015 2233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:17.085776 2238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:17.085752 2241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:17.085909 2239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:17.086725 2233 server_base.cc:1034] running on GCE node
I20260619 06:34:17.087536 2233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:17.088958 2233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:17.090227 2233 hybrid_clock.cc:648] HybridClock initialized: now 1781850857090179 us; error 33 us; skew 500 ppm
I20260619 06:34:17.090641 2233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:17 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850857, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:17.093734 2233 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:34:17.095155 2233 webserver.cc:460] Webserver started at http://127.28.104.254:46809/ using document root <none> and password file <none>
I20260619 06:34:17.095851 2233 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:17.095999 2233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:17.096374 2233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:17.099434 2233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "7f62bb454be94f7689ac79d34ba9cf8c"
format_stamp: "Formatted at 2026-06-19 06:34:17 on dist-test-slave-nf4n"
server_key: "99ab7f85000f5e5c33fd3c0d48ad21f8"
server_key_iv: "7ee178cc0a8f72d8ab0acfdc7d3644e7"
server_key_version: "encryptionkey@0"
I20260619 06:34:17.100272 2233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "7f62bb454be94f7689ac79d34ba9cf8c"
format_stamp: "Formatted at 2026-06-19 06:34:17 on dist-test-slave-nf4n"
server_key: "99ab7f85000f5e5c33fd3c0d48ad21f8"
server_key_iv: "7ee178cc0a8f72d8ab0acfdc7d3644e7"
server_key_version: "encryptionkey@0"
I20260619 06:34:17.105520 2233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260619 06:34:17.108915 2248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:17.110221 2233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:34:17.110421 2233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "7f62bb454be94f7689ac79d34ba9cf8c"
format_stamp: "Formatted at 2026-06-19 06:34:17 on dist-test-slave-nf4n"
server_key: "99ab7f85000f5e5c33fd3c0d48ad21f8"
server_key_iv: "7ee178cc0a8f72d8ab0acfdc7d3644e7"
server_key_version: "encryptionkey@0"
I20260619 06:34:17.110628 2233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:17.134958 2233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:17.139437 2233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:17.139760 2233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:17.158437 2233 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:34677
I20260619 06:34:17.158483 2300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:34677 every 8 connection(s)
I20260619 06:34:17.159879 2233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:17.163537 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 2233
I20260619 06:34:17.163825 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:17.164587 2301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:17.164710 29091 external_mini_cluster.cc:1428] Setting key b38155af2a25747619d7162762870bd2
I20260619 06:34:17.180433 2301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c: Bootstrap starting.
Jun 19 06:34:17 dist-test-slave-nf4n krb5kdc[2219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:17.184048 2301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:17.185326 2301 log.cc:824] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:17.188552 2301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c: No bootstrap required, opened a new log
I20260619 06:34:17.189411 2304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:17.166871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.28.104.254:34677)
0619 06:34:17.167679 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:17.167691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:17.167735 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:34:17.169693 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:17.169716 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:17.170114 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:17.170439 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:17.172464 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:17.173896 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:17.175802 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:17.176546 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:17.176702 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:17.181814 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:17.181822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:17.181838 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:17.181892 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:17.185117 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:17.185898 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:17.185905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:17.185911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:17.186106 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:17.186505 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:17.186512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:17.186523 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:17.187058 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:17.187363 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:17.187818 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 06:34:17.188240 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":368,"threads_started":1}
I20260619 06:34:17.196960 2301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f62bb454be94f7689ac79d34ba9cf8c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34677 } }
I20260619 06:34:17.197464 2301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:17.197584 2301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f62bb454be94f7689ac79d34ba9cf8c, State: Initialized, Role: FOLLOWER
I20260619 06:34:17.198294 2301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f62bb454be94f7689ac79d34ba9cf8c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34677 } }
I20260619 06:34:17.198613 2301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:17.198807 2301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:17.198998 2301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:17.201493 2301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f62bb454be94f7689ac79d34ba9cf8c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34677 } }
I20260619 06:34:17.202049 2301 leader_election.cc:302] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f62bb454be94f7689ac79d34ba9cf8c; no voters:
I20260619 06:34:17.202670 2301 leader_election.cc:288] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:17.202780 2306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:17.203600 2306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [term 1 LEADER]: Becoming Leader. State: Replica: 7f62bb454be94f7689ac79d34ba9cf8c, State: Running, Role: LEADER
I20260619 06:34:17.204234 2306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f62bb454be94f7689ac79d34ba9cf8c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34677 } }
I20260619 06:34:17.205204 2301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:17.207201 2308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f62bb454be94f7689ac79d34ba9cf8c. Latest consensus state: current_term: 1 leader_uuid: "7f62bb454be94f7689ac79d34ba9cf8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f62bb454be94f7689ac79d34ba9cf8c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34677 } } }
I20260619 06:34:17.207623 2308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:17.207464 2307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f62bb454be94f7689ac79d34ba9cf8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f62bb454be94f7689ac79d34ba9cf8c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34677 } } }
I20260619 06:34:17.207916 2307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:17.210107 2312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:17.215808 2312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:17.225590 2312 catalog_manager.cc:1347] Generated new cluster ID: c08ab0aad7c14e1cb31a9a6b1c5d8de8
I20260619 06:34:17.225719 2312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:17.244652 2312 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:17.246140 2312 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:17.253197 2312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f62bb454be94f7689ac79d34ba9cf8c: Generated new TSK 0
I20260619 06:34:17.254117 2312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:34:17.306109 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34677
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:17.632371 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:17.632668 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:17.632761 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:17.640017 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:17.640130 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:17.640240 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:34:17.650178 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:17.651424 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:17.652654 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:17.665663 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:17.665773 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:17.665663 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:17.666812 2327 server_base.cc:1034] running on GCE node
I20260619 06:34:17.667505 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:17.668493 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:17.669737 2327 hybrid_clock.cc:648] HybridClock initialized: now 1781850857669690 us; error 69 us; skew 500 ppm
I20260619 06:34:17.670085 2327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:17 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850857, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:17.673432 2327 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:34:17.674793 2327 webserver.cc:460] Webserver started at http://127.28.104.193:37389/ using document root <none> and password file <none>
I20260619 06:34:17.675506 2327 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:17.675640 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:17.675966 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:17.678905 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "cb6fa54df63940a8a1d3fb84b877be6d"
format_stamp: "Formatted at 2026-06-19 06:34:17 on dist-test-slave-nf4n"
server_key: "969615192d844ca7f8701a9c0c58205d"
server_key_iv: "870a0c7e4aff33a70f926438942b594d"
server_key_version: "encryptionkey@0"
I20260619 06:34:17.679718 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "cb6fa54df63940a8a1d3fb84b877be6d"
format_stamp: "Formatted at 2026-06-19 06:34:17 on dist-test-slave-nf4n"
server_key: "969615192d844ca7f8701a9c0c58205d"
server_key_iv: "870a0c7e4aff33a70f926438942b594d"
server_key_version: "encryptionkey@0"
I20260619 06:34:17.684962 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:34:17.688462 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:17.689893 2327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:17.690056 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "cb6fa54df63940a8a1d3fb84b877be6d"
format_stamp: "Formatted at 2026-06-19 06:34:17 on dist-test-slave-nf4n"
server_key: "969615192d844ca7f8701a9c0c58205d"
server_key_iv: "870a0c7e4aff33a70f926438942b594d"
server_key_version: "encryptionkey@0"
I20260619 06:34:17.690274 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:17.707648 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:17.711864 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:17.712167 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:17.713197 2327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:17.714701 2327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:17.714810 2327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:17.714967 2327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:17.715037 2327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:17.748431 2327 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:42517
I20260619 06:34:17.748514 2455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:42517 every 8 connection(s)
I20260619 06:34:17.749780 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:34:17.756700 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 2327
I20260619 06:34:17.756935 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:34:17.757761 29091 external_mini_cluster.cc:1428] Setting key bcbc3f3307ae668dd25a30b626720a77
Jun 19 06:34:17 dist-test-slave-nf4n krb5kdc[2219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850857, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:17.775034 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:17.752375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:48157 (local address 127.28.104.254:34677)
0619 06:34:17.752900 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:17.752904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:17.754081 (+ 1177us) server_negotiation.cc:407] Connection header received
0619 06:34:17.756045 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:17.756050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:17.756123 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:17.756227 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:17.758523 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:17.759549 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:17.762158 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:17.762598 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:17.762710 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:17.767445 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:17.767455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:17.767461 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:17.767517 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:17.771359 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:17.772101 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:17.772110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:17.772117 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:17.772282 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:17.772695 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:17.772704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:17.772711 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:17.773225 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:17.773405 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:17.774221 (+ 816us) server_negotiation.cc:299] Negotiation successful
0619 06:34:17.774520 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
I20260619 06:34:17.775578 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:17.753097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34677 (local address 127.28.104.193:48157)
0619 06:34:17.753913 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:17.753957 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:17.755777 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:17.756564 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:17.756577 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:17.757416 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:17.758253 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:17.758296 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:17.759766 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:17.759770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:17.761961 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:17.761981 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:17.762848 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:17.762857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:17.763207 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:17.764087 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:17.764129 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:17.767125 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:17.771508 (+ 4383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:17.771516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:17.771530 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:17.771955 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:17.772390 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:17.772393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:17.772395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:17.772565 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:17.773360 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:17.773366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:17.773779 (+ 413us) client_negotiation.cc:769] Sending connection context
0619 06:34:17.774039 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 06:34:17.774447 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":346,"threads_started":1}
I20260619 06:34:17.777241 2456 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34677
I20260619 06:34:17.777592 2456 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:17.778323 2456 heartbeater.cc:511] Master 127.28.104.254:34677 requested a full tablet report, sending...
I20260619 06:34:17.780418 2265 ts_manager.cc:194] Registered new tserver with Master: cb6fa54df63940a8a1d3fb84b877be6d (127.28.104.193:42517)
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
I20260619 06:34:17.781966 2265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:48157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:34:17.808837 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34677
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:18.134984 2462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:18.135325 2462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:18.135509 2462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:18.142678 2462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:18.142836 2462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:18.143028 2462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:34:18.152640 2462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:18.154014 2462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:18.155361 2462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:18.168813 2467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:18.168725 2468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:18.169430 2470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:18.169826 2462 server_base.cc:1034] running on GCE node
I20260619 06:34:18.170526 2462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:18.171620 2462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:18.172855 2462 hybrid_clock.cc:648] HybridClock initialized: now 1781850858172806 us; error 34 us; skew 500 ppm
I20260619 06:34:18.173233 2462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:18.176864 2462 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:34:18.178314 2462 webserver.cc:460] Webserver started at http://127.28.104.194:34095/ using document root <none> and password file <none>
I20260619 06:34:18.178992 2462 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:18.179121 2462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:18.179428 2462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:18.182520 2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "953168ca33804a37838bd407c3728871"
format_stamp: "Formatted at 2026-06-19 06:34:18 on dist-test-slave-nf4n"
server_key: "20068b4f7e88ce95d714c854de26acc7"
server_key_iv: "cf78f290d6bca5c0f539c79b7c7aaf52"
server_key_version: "encryptionkey@0"
I20260619 06:34:18.183326 2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "953168ca33804a37838bd407c3728871"
format_stamp: "Formatted at 2026-06-19 06:34:18 on dist-test-slave-nf4n"
server_key: "20068b4f7e88ce95d714c854de26acc7"
server_key_iv: "cf78f290d6bca5c0f539c79b7c7aaf52"
server_key_version: "encryptionkey@0"
I20260619 06:34:18.188431 2462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:34:18.191866 2477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:18.193222 2462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:18.193408 2462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "953168ca33804a37838bd407c3728871"
format_stamp: "Formatted at 2026-06-19 06:34:18 on dist-test-slave-nf4n"
server_key: "20068b4f7e88ce95d714c854de26acc7"
server_key_iv: "cf78f290d6bca5c0f539c79b7c7aaf52"
server_key_version: "encryptionkey@0"
I20260619 06:34:18.193605 2462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:18.210354 2462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:18.214548 2462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:18.214840 2462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:18.215929 2462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:18.217375 2462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:18.217485 2462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:18.217620 2462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:18.217685 2462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:18.250185 2462 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:34901
I20260619 06:34:18.250226 2590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:34901 every 8 connection(s)
I20260619 06:34:18.251546 2462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:34:18.258805 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 2462
I20260619 06:34:18.258996 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:34:18.259478 29091 external_mini_cluster.cc:1428] Setting key 0a2ca16554a2e4bffd3ee27ef40c86ed
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:18.271929 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:18.254068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:39467 (local address 127.28.104.254:34677)
0619 06:34:18.254249 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:18.254254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:18.255421 (+ 1167us) server_negotiation.cc:407] Connection header received
0619 06:34:18.257157 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:18.257162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:18.257232 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:18.257319 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:18.258760 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.259623 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.261421 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.261843 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.261986 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:18.265579 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:18.265588 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:18.265595 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:18.265642 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:18.268610 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.269614 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.269617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.269620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.269752 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.270226 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.270229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.270231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.270484 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:18.270585 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:18.271497 (+ 912us) server_negotiation.cc:299] Negotiation successful
0619 06:34:18.271673 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:34:18.273550 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:18.254565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34677 (local address 127.28.104.194:39467)
0619 06:34:18.255257 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:18.255295 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:18.256856 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:18.257523 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:18.257531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:18.258049 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:18.258587 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:18.258602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.259769 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.259774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:18.261276 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:18.261288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.262065 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.262072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:18.262319 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:18.263032 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:18.263054 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:18.265338 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:18.268761 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:18.268773 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:18.268795 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:18.269485 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:18.269864 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:18.269870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:18.269874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:18.270117 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:18.270611 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:18.270623 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:18.271074 (+ 451us) client_negotiation.cc:769] Sending connection context
0619 06:34:18.271387 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 06:34:18.271960 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":280,"threads_started":1}
I20260619 06:34:18.275089 2591 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34677
I20260619 06:34:18.275468 2591 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:18.276242 2591 heartbeater.cc:511] Master 127.28.104.254:34677 requested a full tablet report, sending...
I20260619 06:34:18.277796 2265 ts_manager.cc:194] Registered new tserver with Master: 953168ca33804a37838bd407c3728871 (127.28.104.194:34901)
I20260619 06:34:18.278636 2265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:39467
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:34:18.308095 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34677
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:18.632617 2596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:18.632962 2596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:18.633188 2596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:18.640583 2596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:18.640751 2596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:18.640966 2596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:34:18.651167 2596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:18.652577 2596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:18.654044 2596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:18.667907 2602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:18.668015 2601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:18.668020 2604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:18.669003 2596 server_base.cc:1034] running on GCE node
I20260619 06:34:18.669710 2596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:18.670701 2596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:18.671916 2596 hybrid_clock.cc:648] HybridClock initialized: now 1781850858671858 us; error 60 us; skew 500 ppm
I20260619 06:34:18.672266 2596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:18.675451 2596 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:34:18.676937 2596 webserver.cc:460] Webserver started at http://127.28.104.195:33673/ using document root <none> and password file <none>
I20260619 06:34:18.677668 2596 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:18.677817 2596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:18.678129 2596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:18.681171 2596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "29aebe3eb2c149ce9327dd2b3893f4fb"
format_stamp: "Formatted at 2026-06-19 06:34:18 on dist-test-slave-nf4n"
server_key: "28db771190db3163ba774fc0b0649a55"
server_key_iv: "958090ce831ef5b8aaafe98488c037f7"
server_key_version: "encryptionkey@0"
I20260619 06:34:18.681964 2596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "29aebe3eb2c149ce9327dd2b3893f4fb"
format_stamp: "Formatted at 2026-06-19 06:34:18 on dist-test-slave-nf4n"
server_key: "28db771190db3163ba774fc0b0649a55"
server_key_iv: "958090ce831ef5b8aaafe98488c037f7"
server_key_version: "encryptionkey@0"
I20260619 06:34:18.687218 2596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:34:18.690869 2611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:18.692222 2596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:34:18.692386 2596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "29aebe3eb2c149ce9327dd2b3893f4fb"
format_stamp: "Formatted at 2026-06-19 06:34:18 on dist-test-slave-nf4n"
server_key: "28db771190db3163ba774fc0b0649a55"
server_key_iv: "958090ce831ef5b8aaafe98488c037f7"
server_key_version: "encryptionkey@0"
I20260619 06:34:18.692595 2596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:18.711891 2596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:18.716535 2596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:18.716847 2596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:18.717942 2596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:18.719352 2596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:18.719425 2596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:18.719601 2596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:18.719651 2596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:18.753768 2596 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:42597
I20260619 06:34:18.753808 2723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:42597 every 8 connection(s)
I20260619 06:34:18.755044 2596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:34:18.758543 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 2596
I20260619 06:34:18.758754 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:34:18.759325 29091 external_mini_cluster.cc:1428] Setting key 02f15d3bbaf11b49905d65ea9a4eb07f
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:18.776075 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:18.757605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:60759 (local address 127.28.104.254:34677)
0619 06:34:18.757774 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:18.757778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:18.759219 (+ 1441us) server_negotiation.cc:407] Connection header received
0619 06:34:18.761493 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:18.761499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:18.761590 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:18.761705 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:18.763361 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.764103 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.765709 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.766185 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.766327 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:18.769690 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:18.769701 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:18.769704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:18.769737 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:18.772067 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.772830 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.772836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.772840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.773060 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.773592 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.773614 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.773618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.774097 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:18.774262 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:18.775344 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0619 06:34:18.775633 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:34:18.777664 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:18.758104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34677 (local address 127.28.104.195:60759)
0619 06:34:18.758980 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:18.759025 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:18.761146 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:18.761962 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:18.761971 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:18.762631 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:18.763190 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:18.763207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.764257 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.764262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:18.765564 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:18.765575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.766408 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.766417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:18.766655 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:18.767258 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:18.767285 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:18.769488 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:18.772221 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:18.772230 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:18.772248 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:18.772673 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:18.773192 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:18.773197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:18.773200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:18.773448 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:18.774226 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:18.774235 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:18.774808 (+ 573us) client_negotiation.cc:769] Sending connection context
0619 06:34:18.775181 (+ 373us) client_negotiation.cc:240] Negotiation successful
0619 06:34:18.775935 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":406,"threads_started":1}
I20260619 06:34:18.779206 2724 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34677
I20260619 06:34:18.779529 2724 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:18.780438 2724 heartbeater.cc:511] Master 127.28.104.254:34677 requested a full tablet report, sending...
I20260619 06:34:18.782243 2265 ts_manager.cc:194] Registered new tserver with Master: 29aebe3eb2c149ce9327dd2b3893f4fb (127.28.104.195:42597)
I20260619 06:34:18.783363 2265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:60759
I20260619 06:34:18.785104 2456 heartbeater.cc:503] Master 127.28.104.254:34677 was elected leader, sending a full tablet report...
I20260619 06:34:18.787499 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/test-admin.keytab.
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:18.814924 29091 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 06:34:18.821226 2731 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:18.817812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.28.104.193:42517)
0619 06:34:18.819229 (+ 1417us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:18.819236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:18.819263 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:34:18.819359 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:18.819365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:18.819583 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:18.819707 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:18.820734 (+ 1027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34130: BlockingRecv error: recv got EOF from 127.0.0.1:34130 (error 108)
Metrics: {"server-negotiator.queue_time_us":1256,"thread_start_us":1093,"threads_started":1}
W20260619 06:34:18.821368 29091 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:18.823120 29091 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:18.824738 29091 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:34:18.840190 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:18.826583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.28.104.193:42517)
0619 06:34:18.826784 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:18.826791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:18.826962 (+ 171us) server_negotiation.cc:407] Connection header received
0619 06:34:18.827112 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:18.827115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:18.827174 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:18.827257 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:18.828202 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.829229 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.830375 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.830937 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.831030 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:18.834713 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:18.834725 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:18.834740 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:18.834780 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:18.837714 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.838323 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.838330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.838332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.838449 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.838753 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.838756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.838758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.839106 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:18.839316 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:18.839636 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 06:34:18.839914 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 06:34:18.846056 2731 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:18.844227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.28.104.193:42517)
0619 06:34:18.844370 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:18.844376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:18.844744 (+ 368us) server_negotiation.cc:407] Connection header received
0619 06:34:18.844864 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:18.844867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:18.844936 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:18.845011 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:18.845785 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34148: BlockingRecv error: recv got EOF from 127.0.0.1:34148 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 06:34:18.846529 29091 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5cc)
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:18.848019 29091 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:18.849507 29091 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:34:18.863808 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:18.851207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.28.104.193:42517)
0619 06:34:18.851439 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:18.851445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:18.851659 (+ 214us) server_negotiation.cc:407] Connection header received
0619 06:34:18.851769 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:18.851772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:18.851829 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:18.851904 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:18.852731 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.853539 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.854705 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.855151 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.855288 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:18.858797 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:18.858802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:18.858805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:18.858836 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:18.861301 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.862153 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.862159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.862163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.862290 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.862659 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.862663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.862665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.862944 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:18.863129 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:18.863432 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 06:34:18.863588 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 06:34:18.868985 2731 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:18.867111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.28.104.193:42517)
0619 06:34:18.867321 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:18.867328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:18.867555 (+ 227us) server_negotiation.cc:407] Connection header received
0619 06:34:18.867739 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:18.867744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:18.867824 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:18.867929 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:18.868747 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34180: BlockingRecv error: recv got EOF from 127.0.0.1:34180 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 06:34:18.869261 29091 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850805520649-29091-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:18.870730 29091 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:18.872092 29091 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:34:18 dist-test-slave-nf4n krb5kdc[2219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:34:18.886466 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:18.873511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.28.104.193:42517)
0619 06:34:18.873720 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:18.873728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:18.874026 (+ 298us) server_negotiation.cc:407] Connection header received
0619 06:34:18.874203 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:18.874209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:18.874279 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:18.874418 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:18.875315 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.875997 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.877296 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.877719 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.877847 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:18.881425 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:18.881430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:18.881433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:18.881463 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:18.884117 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.884734 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.884739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.884741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.884889 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.885269 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.885273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.885275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.885559 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:18.885774 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:18.886009 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:34:18.886167 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:34:18.887984 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 2327
I20260619 06:34:18.899461 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:18.888629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.28.104.193:42517)
0619 06:34:18.888782 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:18.888785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:18.889198 (+ 413us) server_negotiation.cc:407] Connection header received
0619 06:34:18.889329 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:18.889332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:18.889393 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:18.889465 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:18.890292 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.890942 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.892101 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:18.892508 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:18.892599 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:18.894720 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:18.894724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:18.894728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:18.894755 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:18.897261 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.897892 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.897896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.897899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.898021 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:18.898322 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:18.898325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:18.898326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:18.898565 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:18.898767 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:18.899053 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 06:34:18.899202 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:34:18.900051 2451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:18.982595 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 2327
I20260619 06:34:18.999352 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 2462
Jun 19 06:34:19 dist-test-slave-nf4n krb5kdc[2219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:34:19.015551 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:19.000279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44434 (local address 127.28.104.194:34901)
0619 06:34:19.000852 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:19.000859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:19.000886 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:34:19.000954 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:19.000960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:19.001249 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:19.001398 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:19.002307 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:19.003326 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:19.004535 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:19.005261 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:19.005358 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:19.009489 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:19.009501 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:19.009516 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:19.009547 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:19.012646 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:19.013455 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:19.013471 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:19.013473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:19.013598 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:19.013999 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:19.014002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:19.014004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:19.014401 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:19.014668 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:19.015010 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 06:34:19.015231 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":302,"threads_started":1}
I20260619 06:34:19.016775 2585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:19.098481 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 2462
I20260619 06:34:19.114336 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 2596
Jun 19 06:34:19 dist-test-slave-nf4n krb5kdc[2219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:34:19.131235 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:19.115595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.28.104.195:42597)
0619 06:34:19.116233 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:19.116240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:19.116267 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:34:19.116329 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:19.116335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:19.116551 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:19.116680 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:19.118034 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:19.119439 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:19.120812 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:19.121521 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:19.121620 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:19.126016 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:19.126027 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:19.126041 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:19.126074 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:19.128684 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:19.129428 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:19.129434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:19.129436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:19.129553 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:19.129895 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:19.129898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:19.129900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:19.130217 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:19.130451 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:19.130684 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 06:34:19.130933 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":334,"threads_started":1}
I20260619 06:34:19.132508 2719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:19.217367 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 2596
I20260619 06:34:19.233335 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 2233
I20260619 06:34:19.234638 2295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:19.340564 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 2233
2026-06-19T06:34:19Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3946 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:34:19 dist-test-slave-nf4n krb5kdc[2752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:34:19 dist-test-slave-nf4n krb5kdc[2752](info): set up 2 sockets
Jun 19 06:34:19 dist-test-slave-nf4n krb5kdc[2752](info): commencing operation
krb5kdc: starting...
W20260619 06:34:20.442247 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:34:20 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.254.keytab.
Entry for principal oryx/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.254.keytab.
I20260619 06:34:20.653614 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:35203
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.254.keytab
--principal=oryx/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:35203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:20.979938 2765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:20.980324 2765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:20.980486 2765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:20.987946 2765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:20.988114 2765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:20.988238 2765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:20.988351 2765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:20.988430 2765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:20.998734 2765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:35203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.254.keytab
--principal=oryx/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:35203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:21.000198 2765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:21.001631 2765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:21.014230 2771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:21.014199 2773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:21.014679 2770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:21.015012 2765 server_base.cc:1034] running on GCE node
I20260619 06:34:21.015894 2765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:21.017535 2765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:21.018776 2765 hybrid_clock.cc:648] HybridClock initialized: now 1781850861018743 us; error 56 us; skew 500 ppm
I20260619 06:34:21.019147 2765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:21 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850861, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:21.027527 2765 init.cc:377] Logged in from keytab as oryx/127.28.104.254@KRBTEST.COM (short username oryx)
I20260619 06:34:21.029337 2765 webserver.cc:460] Webserver started at http://127.28.104.254:33829/ using document root <none> and password file <none>
I20260619 06:34:21.030084 2765 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:21.030241 2765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:21.030570 2765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:21.033780 2765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "573e48de9df3491b85efe17e27d6a911"
format_stamp: "Formatted at 2026-06-19 06:34:21 on dist-test-slave-nf4n"
server_key: "20946bd2a94b4a252ee6fc53788d07a6"
server_key_iv: "23ab72385e5f671e8a58c0c5b69629d4"
server_key_version: "encryptionkey@0"
I20260619 06:34:21.034577 2765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "573e48de9df3491b85efe17e27d6a911"
format_stamp: "Formatted at 2026-06-19 06:34:21 on dist-test-slave-nf4n"
server_key: "20946bd2a94b4a252ee6fc53788d07a6"
server_key_iv: "23ab72385e5f671e8a58c0c5b69629d4"
server_key_version: "encryptionkey@0"
I20260619 06:34:21.039839 2765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:34:21.043478 2780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:21.044979 2765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:34:21.045207 2765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "573e48de9df3491b85efe17e27d6a911"
format_stamp: "Formatted at 2026-06-19 06:34:21 on dist-test-slave-nf4n"
server_key: "20946bd2a94b4a252ee6fc53788d07a6"
server_key_iv: "23ab72385e5f671e8a58c0c5b69629d4"
server_key_version: "encryptionkey@0"
I20260619 06:34:21.045439 2765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:21.061302 2765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:21.065496 2765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:21.065789 2765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:21.084245 2765 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:35203
I20260619 06:34:21.084268 2831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:35203 every 8 connection(s)
I20260619 06:34:21.085783 2765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:21.089720 2832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:21.093950 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 2765
I20260619 06:34:21.094187 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:21.094805 29091 external_mini_cluster.cc:1428] Setting key 0abe41f88361600f04ccd67952a72d8c
I20260619 06:34:21.102823 2832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911: Bootstrap starting.
I20260619 06:34:21.106863 2832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:21.108300 2832 log.cc:824] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:34:21 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:21.111802 2832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911: No bootstrap required, opened a new log
I20260619 06:34:21.120479 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:21.096813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.28.104.254:35203)
0619 06:34:21.097725 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:21.097755 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:21.097813 (+ 58us) server_negotiation.cc:407] Connection header received
0619 06:34:21.100912 (+ 3099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:21.100944 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:21.101496 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:21.101973 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:21.103330 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:21.105423 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:21.106625 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:21.107802 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:21.108023 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:21.112763 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:21.112772 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:21.112788 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:21.112826 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:21.116091 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:21.116991 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:21.116997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:21.117004 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:21.117236 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:21.117694 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:21.117697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:21.117699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:21.118253 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:21.118479 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:21.118987 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 06:34:21.119371 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":387,"threads_started":1}
I20260619 06:34:21.121917 2832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } }
I20260619 06:34:21.122337 2832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:21.122448 2832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573e48de9df3491b85efe17e27d6a911, State: Initialized, Role: FOLLOWER
I20260619 06:34:21.123107 2832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } }
I20260619 06:34:21.123392 2832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:21.123579 2832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:21.123767 2832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:21.126217 2832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } }
I20260619 06:34:21.126811 2832 leader_election.cc:302] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 573e48de9df3491b85efe17e27d6a911; no voters:
I20260619 06:34:21.127496 2832 leader_election.cc:288] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:21.127592 2837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:21.128464 2837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 1 LEADER]: Becoming Leader. State: Replica: 573e48de9df3491b85efe17e27d6a911, State: Running, Role: LEADER
I20260619 06:34:21.129060 2837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } }
I20260619 06:34:21.130105 2832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:21.132205 2839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 573e48de9df3491b85efe17e27d6a911. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } }
I20260619 06:34:21.132663 2839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:21.132277 2838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } }
I20260619 06:34:21.133005 2838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:21.142366 2842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:21.147974 2842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:21.157727 2842 catalog_manager.cc:1347] Generated new cluster ID: 7951725ab48a4ccfae5b607186e2a0c7
I20260619 06:34:21.157845 2842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:21.180060 2842 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:21.181561 2842 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:21.187795 2842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911: Generated new TSK 0
I20260619 06:34:21.188643 2842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab.
Entry for principal oryx/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab.
I20260619 06:34:21.249225 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35203
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab
--principal=oryx/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:21.593779 2858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:21.594101 2858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:21.594274 2858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:21.601653 2858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:21.601801 2858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:21.601986 2858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:34:21.612028 2858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab
--principal=oryx/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:21.613497 2858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:21.614877 2858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:21.628037 2863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:21.628067 2864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:21.628093 2866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:21.628850 2858 server_base.cc:1034] running on GCE node
I20260619 06:34:21.629557 2858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:21.630532 2858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:21.631800 2858 hybrid_clock.cc:648] HybridClock initialized: now 1781850861631733 us; error 33 us; skew 500 ppm
I20260619 06:34:21.632146 2858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:21 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850861, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:21.635468 2858 init.cc:377] Logged in from keytab as oryx/127.28.104.193@KRBTEST.COM (short username oryx)
I20260619 06:34:21.636929 2858 webserver.cc:460] Webserver started at http://127.28.104.193:35623/ using document root <none> and password file <none>
I20260619 06:34:21.637660 2858 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:21.637759 2858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:21.638007 2858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:21.640918 2858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "4fad813d2f6144bda0cba9417d0af8b8"
format_stamp: "Formatted at 2026-06-19 06:34:21 on dist-test-slave-nf4n"
server_key: "41e5c8460cfd4ec4a78b41a8c5dab719"
server_key_iv: "426ae9a3a5a70f5a73a351f5696e8073"
server_key_version: "encryptionkey@0"
I20260619 06:34:21.641729 2858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "4fad813d2f6144bda0cba9417d0af8b8"
format_stamp: "Formatted at 2026-06-19 06:34:21 on dist-test-slave-nf4n"
server_key: "41e5c8460cfd4ec4a78b41a8c5dab719"
server_key_iv: "426ae9a3a5a70f5a73a351f5696e8073"
server_key_version: "encryptionkey@0"
I20260619 06:34:21.647063 2858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:34:21.650631 2873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:21.652133 2858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:34:21.652333 2858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "4fad813d2f6144bda0cba9417d0af8b8"
format_stamp: "Formatted at 2026-06-19 06:34:21 on dist-test-slave-nf4n"
server_key: "41e5c8460cfd4ec4a78b41a8c5dab719"
server_key_iv: "426ae9a3a5a70f5a73a351f5696e8073"
server_key_version: "encryptionkey@0"
I20260619 06:34:21.652546 2858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:21.671447 2858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:21.676350 2858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:21.676700 2858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:21.677778 2858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:21.679222 2858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:21.679312 2858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:21.679498 2858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:21.679565 2858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:21.714946 2858 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:37853
I20260619 06:34:21.714993 2988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:37853 every 8 connection(s)
I20260619 06:34:21.716403 2858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:34:21.723675 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 2858
I20260619 06:34:21.723966 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:34:21.724638 29091 external_mini_cluster.cc:1428] Setting key 6bcfe26c26d764ee8da16b82eff09d33
Jun 19 06:34:21 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850861, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:21.738879 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:21.719247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:57015 (local address 127.28.104.254:35203)
0619 06:34:21.719725 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:21.719730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:21.720591 (+ 861us) server_negotiation.cc:407] Connection header received
0619 06:34:21.722547 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:21.722553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:21.722649 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:21.722772 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:21.724484 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:21.725383 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:21.727614 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:21.728015 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:21.728115 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:21.732522 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:21.732531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:21.732539 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:21.732586 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:21.735468 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:21.736450 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:21.736455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:21.736457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:21.736584 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:21.737119 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:21.737122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:21.737124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:21.737387 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:21.737500 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:21.738430 (+ 930us) server_negotiation.cc:299] Negotiation successful
0619 06:34:21.738612 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":261,"threads_started":1}
I20260619 06:34:21.740510 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:21.719690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.193:57015)
0619 06:34:21.720422 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:21.720470 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:21.722257 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:21.723037 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:21.723046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:21.723605 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:21.724305 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:21.724325 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:21.725625 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:21.725630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:21.727459 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:21.727476 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:21.728260 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:21.728268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:21.728616 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:21.729475 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:21.729513 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:21.732288 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:21.735616 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:21.735629 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:21.735651 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:21.736322 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:21.736701 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:21.736708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:21.736715 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:21.736977 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:21.737516 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:21.737527 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:21.738014 (+ 487us) client_negotiation.cc:769] Sending connection context
0619 06:34:21.738321 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 06:34:21.738966 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":280,"threads_started":1}
I20260619 06:34:21.742300 2989 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35203
I20260619 06:34:21.742646 2989 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:21.743621 2989 heartbeater.cc:511] Master 127.28.104.254:35203 requested a full tablet report, sending...
I20260619 06:34:21.746304 2797 ts_manager.cc:194] Registered new tserver with Master: 4fad813d2f6144bda0cba9417d0af8b8 (127.28.104.193:37853)
WARNING: no policy specified for oryx/127.28.104.194@KRBTEST.COM; defaulting to no policy
I20260619 06:34:21.748548 2797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:57015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab.
Entry for principal oryx/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab.
I20260619 06:34:21.775871 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35203
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab
--principal=oryx/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:22.098985 2995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:22.099251 2995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:22.099344 2995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:22.106735 2995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:22.106848 2995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:22.106961 2995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:34:22.117000 2995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab
--principal=oryx/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:22.118328 2995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:22.119616 2995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:22.132864 3000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:22.132913 3003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:22.132794 3001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:22.133738 2995 server_base.cc:1034] running on GCE node
I20260619 06:34:22.134385 2995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:22.135350 2995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:22.136582 2995 hybrid_clock.cc:648] HybridClock initialized: now 1781850862136529 us; error 47 us; skew 500 ppm
I20260619 06:34:22.136909 2995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:22 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850862, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:22.140079 2995 init.cc:377] Logged in from keytab as oryx/127.28.104.194@KRBTEST.COM (short username oryx)
I20260619 06:34:22.141637 2995 webserver.cc:460] Webserver started at http://127.28.104.194:43217/ using document root <none> and password file <none>
I20260619 06:34:22.142297 2995 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:22.142396 2995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:22.142633 2995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:22.145794 2995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "c43f378aaf1940cc800b33e299cde2aa"
format_stamp: "Formatted at 2026-06-19 06:34:22 on dist-test-slave-nf4n"
server_key: "e0069f27255383834aa60c15e80f6629"
server_key_iv: "d78cfa4627c2bfa1d475dddca616f5e4"
server_key_version: "encryptionkey@0"
I20260619 06:34:22.146538 2995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "c43f378aaf1940cc800b33e299cde2aa"
format_stamp: "Formatted at 2026-06-19 06:34:22 on dist-test-slave-nf4n"
server_key: "e0069f27255383834aa60c15e80f6629"
server_key_iv: "d78cfa4627c2bfa1d475dddca616f5e4"
server_key_version: "encryptionkey@0"
I20260619 06:34:22.151657 2995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:34:22.155407 3010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:22.156699 2995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:34:22.156870 2995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "c43f378aaf1940cc800b33e299cde2aa"
format_stamp: "Formatted at 2026-06-19 06:34:22 on dist-test-slave-nf4n"
server_key: "e0069f27255383834aa60c15e80f6629"
server_key_iv: "d78cfa4627c2bfa1d475dddca616f5e4"
server_key_version: "encryptionkey@0"
I20260619 06:34:22.157090 2995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:22.183781 2995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:22.187779 2995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:22.188086 2995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:22.189119 2995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:22.190455 2995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:22.190527 2995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:22.190672 2995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:22.190738 2995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:22.223259 2995 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:45547
I20260619 06:34:22.223317 3123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:45547 every 8 connection(s)
I20260619 06:34:22.224613 2995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:34:22.225975 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 2995
I20260619 06:34:22.226234 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:34:22.226696 29091 external_mini_cluster.cc:1428] Setting key ca2cb50d0f79a9a9608c263fc2254c03
Jun 19 06:34:22 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850862, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:22.247014 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:22.227367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:34931 (local address 127.28.104.254:35203)
0619 06:34:22.227554 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:22.227558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:22.229145 (+ 1587us) server_negotiation.cc:407] Connection header received
0619 06:34:22.231148 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:22.231155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:22.231295 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:22.231415 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:22.232987 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:22.233761 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:22.235334 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:22.235831 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:22.235970 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:22.239665 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:22.239674 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:22.239682 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:22.239728 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:22.243533 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:22.244322 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:22.244329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:22.244336 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:22.244531 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:22.244969 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:22.244977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:22.244982 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:22.245420 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:22.245577 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:22.246311 (+ 734us) server_negotiation.cc:299] Negotiation successful
0619 06:34:22.246592 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for oryx/127.28.104.195@KRBTEST.COM; defaulting to no policy
I20260619 06:34:22.247737 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:22.228034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.194:34931)
0619 06:34:22.228951 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:22.228996 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:22.230878 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:22.231679 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:22.231693 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:22.232227 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:22.232821 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:22.232838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:22.233925 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:22.233937 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:22.235173 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:22.235190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:22.236060 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:22.236066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:22.236305 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:22.237104 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:22.237130 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:22.239414 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:22.243694 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:22.243701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:22.243715 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:22.244186 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:22.244655 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:22.244658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:22.244661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:22.244854 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:22.245542 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:22.245548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:22.245902 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 06:34:22.246135 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 06:34:22.246568 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":375,"threads_started":1}
I20260619 06:34:22.249435 3124 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35203
I20260619 06:34:22.249852 3124 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.195@KRBTEST.COM" created.
I20260619 06:34:22.250989 3124 heartbeater.cc:511] Master 127.28.104.254:35203 requested a full tablet report, sending...
I20260619 06:34:22.252844 2797 ts_manager.cc:194] Registered new tserver with Master: c43f378aaf1940cc800b33e299cde2aa (127.28.104.194:45547)
I20260619 06:34:22.254165 2797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:34931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab.
Entry for principal oryx/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab.
I20260619 06:34:22.276371 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35203
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab
--principal=oryx/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:22.603171 3129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:22.603468 3129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:22.603628 3129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:22.610847 3129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:22.610965 3129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:22.611158 3129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:34:22.621260 3129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab
--principal=oryx/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:35203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:22.622550 3129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:22.623869 3129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:22.637130 3134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:22.637207 3137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:22.637107 3135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:22.638010 3129 server_base.cc:1034] running on GCE node
I20260619 06:34:22.638634 3129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:22.639504 3129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:22.640700 3129 hybrid_clock.cc:648] HybridClock initialized: now 1781850862640637 us; error 54 us; skew 500 ppm
I20260619 06:34:22.640976 3129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:22 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850862, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:22.644455 3129 init.cc:377] Logged in from keytab as oryx/127.28.104.195@KRBTEST.COM (short username oryx)
I20260619 06:34:22.646050 3129 webserver.cc:460] Webserver started at http://127.28.104.195:40085/ using document root <none> and password file <none>
I20260619 06:34:22.646780 3129 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:22.646922 3129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:22.647235 3129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:22.650450 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "e688da4658c34574bc5a3f2a7f01417f"
format_stamp: "Formatted at 2026-06-19 06:34:22 on dist-test-slave-nf4n"
server_key: "dc18c473a8627a9725fbbc3ad2e8cba8"
server_key_iv: "a60b7be7fa81e645842972f39d2e3ddf"
server_key_version: "encryptionkey@0"
I20260619 06:34:22.651254 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "e688da4658c34574bc5a3f2a7f01417f"
format_stamp: "Formatted at 2026-06-19 06:34:22 on dist-test-slave-nf4n"
server_key: "dc18c473a8627a9725fbbc3ad2e8cba8"
server_key_iv: "a60b7be7fa81e645842972f39d2e3ddf"
server_key_version: "encryptionkey@0"
I20260619 06:34:22.656651 3129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:34:22.660427 3144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:22.661914 3129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:22.662088 3129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "e688da4658c34574bc5a3f2a7f01417f"
format_stamp: "Formatted at 2026-06-19 06:34:22 on dist-test-slave-nf4n"
server_key: "dc18c473a8627a9725fbbc3ad2e8cba8"
server_key_iv: "a60b7be7fa81e645842972f39d2e3ddf"
server_key_version: "encryptionkey@0"
I20260619 06:34:22.662302 3129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:22.676762 3129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:22.680691 3129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:22.680989 3129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:22.681949 3129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:22.683290 3129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:22.683364 3129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:22.683542 3129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:22.683604 3129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:22.717545 3129 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:41285
I20260619 06:34:22.717579 3257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:41285 every 8 connection(s)
I20260619 06:34:22.718931 3129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:34:22.726910 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 3129
I20260619 06:34:22.727101 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:34:22.727592 29091 external_mini_cluster.cc:1428] Setting key f632ee59824850bd0fd19610f8c2e182
Jun 19 06:34:22 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850862, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:22.738668 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:22.721650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:37985 (local address 127.28.104.254:35203)
0619 06:34:22.721843 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:22.721848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:22.722961 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 06:34:22.724636 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:22.724641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:22.724734 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:22.724823 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:22.726304 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:22.727122 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:22.729070 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:22.729477 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:22.729612 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:22.733273 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:22.733278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:22.733281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:22.733312 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:22.735598 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:22.736315 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:22.736320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:22.736324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:22.736457 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:22.736868 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:22.736876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:22.736879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:22.737239 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:22.737391 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:22.738087 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 06:34:22.738312 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:34:22.739646 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:22.722083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.195:37985)
0619 06:34:22.722799 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:22.722844 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:22.724396 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:22.725086 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:22.725103 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:22.725589 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:22.726139 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:22.726155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:22.727316 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:22.727320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:22.728911 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:22.728923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:22.729697 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:22.729704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:22.730007 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:22.730763 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:22.730785 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:22.733068 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:22.735757 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:22.735765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:22.735782 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:22.736189 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:22.736568 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:22.736573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:22.736577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:22.736762 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:22.737371 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:22.737380 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:22.737734 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 06:34:22.737994 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 06:34:22.738553 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":276,"threads_started":1}
I20260619 06:34:22.741073 3258 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35203
I20260619 06:34:22.741412 3258 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:22.742394 3258 heartbeater.cc:511] Master 127.28.104.254:35203 requested a full tablet report, sending...
I20260619 06:34:22.743857 2797 ts_manager.cc:194] Registered new tserver with Master: e688da4658c34574bc5a3f2a7f01417f (127.28.104.195:41285)
I20260619 06:34:22.744673 2797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:37985
I20260619 06:34:22.751904 2989 heartbeater.cc:503] Master 127.28.104.254:35203 was elected leader, sending a full tablet report...
I20260619 06:34:22.755364 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.104.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.253.keytab.
Entry for principal oryx/127.28.104.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.253.keytab.
I20260619 06:34:22.809551 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.253:46139
--webserver_interface=127.28.104.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.253.keytab
--principal=oryx/127.28.104.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:35203,127.28.104.253:46139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:23.139184 3263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:23.139535 3263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:23.139693 3263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:23.146955 3263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:23.147104 3263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:23.147208 3263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:23.147308 3263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:23.147382 3263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:23.157420 3263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:35203,127.28.104.253:46139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.253.keytab
--principal=oryx/127.28.104.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.253:46139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.104.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
W20260619 06:34:23.157827 3263 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.104.254:35203,127.28.104.253:46139'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 06:34:23.158907 3263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:23.160349 3263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:23.172838 3271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:23.172838 3269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:23.172874 3268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:23.173537 3263 server_base.cc:1034] running on GCE node
I20260619 06:34:23.174301 3263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:23.175724 3263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:23.176962 3263 hybrid_clock.cc:648] HybridClock initialized: now 1781850863176898 us; error 42 us; skew 500 ppm
I20260619 06:34:23.177341 3263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850863, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:23.180749 3263 init.cc:377] Logged in from keytab as oryx/127.28.104.253@KRBTEST.COM (short username oryx)
I20260619 06:34:23.182451 3263 webserver.cc:460] Webserver started at http://127.28.104.253:36751/ using document root <none> and password file <none>
I20260619 06:34:23.183184 3263 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:23.183331 3263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:23.183660 3263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:23.186879 3263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data/instance:
uuid: "d9102c0e7dfe4c00bae643db3ddc784f"
format_stamp: "Formatted at 2026-06-19 06:34:23 on dist-test-slave-nf4n"
server_key: "da1fc4809ba4c14e27394c23e083d23e"
server_key_iv: "e67b2d010e3f7bf528db15619e0be0e4"
server_key_version: "encryptionkey@0"
I20260619 06:34:23.187674 3263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal/instance:
uuid: "d9102c0e7dfe4c00bae643db3ddc784f"
format_stamp: "Formatted at 2026-06-19 06:34:23 on dist-test-slave-nf4n"
server_key: "da1fc4809ba4c14e27394c23e083d23e"
server_key_iv: "e67b2d010e3f7bf528db15619e0be0e4"
server_key_version: "encryptionkey@0"
I20260619 06:34:23.193099 3263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:34:23.196642 3278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:23.198165 3263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:34:23.198403 3263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal
uuid: "d9102c0e7dfe4c00bae643db3ddc784f"
format_stamp: "Formatted at 2026-06-19 06:34:23 on dist-test-slave-nf4n"
server_key: "da1fc4809ba4c14e27394c23e083d23e"
server_key_iv: "e67b2d010e3f7bf528db15619e0be0e4"
server_key_version: "encryptionkey@0"
I20260619 06:34:23.198638 3263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:23.214985 3263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:23.219560 3263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:23.219846 3263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:23.238988 3263 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.253:46139
I20260619 06:34:23.239037 3329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.253:46139 every 8 connection(s)
I20260619 06:34:23.240401 3263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
I20260619 06:34:23.244215 3330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:23.250452 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 3263
I20260619 06:34:23.250679 3330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:34:23.257908 3124 heartbeater.cc:503] Master 127.28.104.254:35203 was elected leader, sending a full tablet report...
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850863, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.253@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.253@KRBTEST.COM
I20260619 06:34:23.271602 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.253495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.28.104.254:35203)
0619 06:34:23.253909 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.253913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.254536 (+ 623us) server_negotiation.cc:407] Connection header received
0619 06:34:23.255801 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.255807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.255903 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.256038 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.257530 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.258593 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.260642 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.261159 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.261284 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.265050 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.265058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.265063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.265110 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.268082 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.269060 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.269068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.269073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.269200 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.269679 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.269682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.269684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.270064 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.270201 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.271156 (+ 955us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.271320 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":229,"threads_started":1}
I20260619 06:34:23.273003 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.252591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.28.104.253:46139)
0619 06:34:23.253420 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.253441 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.253499 (+ 58us) server_negotiation.cc:407] Connection header received
0619 06:34:23.255544 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.255565 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.255799 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.256016 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.257724 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.259255 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.260984 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.261840 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.261970 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.266396 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.266409 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.266426 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.266462 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.269850 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.270583 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.270590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.270593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.270742 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.271112 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.271116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.271119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.271483 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.272087 (+ 604us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.272262 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.272483 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":367,"threads_started":1}
I20260619 06:34:23.273078 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.253580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:53118)
0619 06:34:23.254387 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:23.254420 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:23.255561 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:23.256160 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:23.256169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:23.256813 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:23.257365 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.257386 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.258709 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.258715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.260435 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.260451 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.261369 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.261375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.261541 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.262350 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:23.262395 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:23.264830 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:23.268211 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.268222 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.268234 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.268911 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.269312 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.269325 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.269332 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.269566 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.270199 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:23.270205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:23.270704 (+ 499us) client_negotiation.cc:769] Sending connection context
0619 06:34:23.271025 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 06:34:23.271555 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":462,"threads_started":1}
I20260619 06:34:23.274866 3330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850863, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.253@KRBTEST.COM for oryx/127.28.104.253@KRBTEST.COM
I20260619 06:34:23.287060 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.275704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:46139 (local address 127.0.0.1:43518)
0619 06:34:23.275867 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:23.275885 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:23.275983 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:23.276215 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:23.276218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:23.276566 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:23.276694 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.276701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.277728 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.277732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.278646 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.278658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.279259 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.279262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.279425 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.279895 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:23.279910 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:23.281859 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:23.284705 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.284711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.284715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.285135 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.285464 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.285469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.285472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.285630 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.286142 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:23.286146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:23.286279 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 06:34:23.286409 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 06:34:23.286701 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 06:34:23.287133 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.275612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.28.104.253:46139)
0619 06:34:23.275807 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.275813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.275953 (+ 140us) server_negotiation.cc:407] Connection header received
0619 06:34:23.276054 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.276057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.276109 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.276182 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.276800 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.277577 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.278762 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.279140 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.279234 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.282074 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.282079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.282082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.282108 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.284562 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.285234 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.285242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.285245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.285354 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.285726 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.285733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.285738 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.286021 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.286164 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.286489 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.286835 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":57}
I20260619 06:34:23.291788 3330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: Bootstrap starting.
I20260619 06:34:23.294938 3330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:23.295946 3330 log.cc:824] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:23.298498 3330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: No bootstrap required, opened a new log
I20260619 06:34:23.307374 3330 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } }
I20260619 06:34:23.307741 3330 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:23.307862 3330 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9102c0e7dfe4c00bae643db3ddc784f, State: Initialized, Role: FOLLOWER
I20260619 06:34:23.308531 3330 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } }
I20260619 06:34:23.309438 3337 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } } }
I20260619 06:34:23.309713 3337 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:34:23.310488 3330 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:34:23.318948 3348 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:34:23.319061 3348 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:34:23.321834 3263 master_runner.cc:428] Detected that this master d9102c0e7dfe4c00bae643db3ddc784f is joining an existing cluster
I20260619 06:34:23.321955 3263 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.104.253:46139
I20260619 06:34:23.346549 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.326154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:46139 (local address 127.0.0.1:43520)
0619 06:34:23.331442 (+ 5288us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:23.331465 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:23.331979 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:23.332485 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:23.332492 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:23.333280 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:23.333778 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.333794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.335356 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.335364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.336845 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.336857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.338138 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.338145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.338340 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.339210 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:23.339239 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:23.340102 (+ 863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:23.343664 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.343671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.343675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.344051 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.344468 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.344474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.344477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.344604 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.345389 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:23.345394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:23.345571 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 06:34:23.345733 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 06:34:23.346047 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5117,"thread_start_us":11018,"threads_started":1}
I20260619 06:34:23.347476 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.325933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.28.104.253:46139)
0619 06:34:23.326138 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.326147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.331573 (+ 5426us) server_negotiation.cc:407] Connection header received
0619 06:34:23.332125 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.332133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.332208 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.332271 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.333939 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.335145 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.336992 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.337779 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.337889 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.340286 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.340290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.340293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.340320 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.343548 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.344169 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.344177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.344181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.344354 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.344740 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.344746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.344750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.345165 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.345335 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.346859 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.347078 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"spinlock_wait_cycles":22912}
I20260619 06:34:23.354146 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.325921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.28.104.254:35203)
0619 06:34:23.326132 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.326139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.337585 (+ 11446us) server_negotiation.cc:407] Connection header received
0619 06:34:23.337844 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.337851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.337963 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.338038 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.339565 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.340642 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.342813 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.343549 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.343686 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.346500 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.346508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.346516 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.346562 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.350815 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.351686 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.351692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.351696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.351852 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.352328 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.352333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.352337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.352761 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.352920 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.353478 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.353780 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:34:23.354568 3354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.325959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:53124)
0619 06:34:23.337479 (+ 11520us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:23.337500 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:23.337688 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:23.338255 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:23.338259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:23.339013 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:23.339373 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.339388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.340816 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.340821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.342663 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.342679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.343964 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.343972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.344209 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.344922 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:23.344951 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:23.346268 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:23.350954 (+ 4686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.350960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.350965 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.351533 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.351979 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.351986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.351992 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.352184 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.352909 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:23.352915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:23.353171 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 06:34:23.353327 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 06:34:23.353734 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10952,"thread_start_us":5177,"threads_started":1}
WARNING: no policy specified for oryx/127.28.104.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.252@KRBTEST.COM" created.
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850861, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.254@KRBTEST.COM for oryx/127.28.104.253@KRBTEST.COM
I20260619 06:34:23.379951 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.361496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:46139 (local address 127.0.0.1:43532)
0619 06:34:23.362244 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:23.362296 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:23.362547 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:23.363087 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:23.363096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:23.363828 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:23.364059 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.364070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.365197 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.365203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.367139 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.367159 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.368492 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.368499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.368700 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.369487 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:23.369521 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:23.372354 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:23.376533 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.376552 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.376562 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.377255 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.377723 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.377732 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.377739 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.377994 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.378655 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:23.378664 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:23.378928 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 06:34:23.379181 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 06:34:23.379491 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":349,"threads_started":1}
I20260619 06:34:23.379988 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.361496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.28.104.253:46139)
0619 06:34:23.361703 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.361714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.362390 (+ 676us) server_negotiation.cc:407] Connection header received
0619 06:34:23.362805 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.362809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.362904 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.363046 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.364240 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.364977 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.367388 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.368124 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.368263 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.372588 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.372595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.372601 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.372640 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.376355 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.377393 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.377400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.377404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.377585 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.378131 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.378137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.378142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.378497 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.378643 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.379324 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.379594 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:34:23.382544 2797 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.104.253:46139: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f"
member_type: NON_VOTER
last_known_addr {
host: "127.28.104.253"
port: 46139
}
attrs {
promote: true
}
}
dest_uuid: "573e48de9df3491b85efe17e27d6a911"
cas_config_opid_index: -1
I20260619 06:34:23.383826 2797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.252.keytab.
Entry for principal oryx/127.28.104.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.252.keytab.
I20260619 06:34:23.388334 3305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:23.388392 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.252:33477
--webserver_interface=127.28.104.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.252.keytab
--principal=oryx/127.28.104.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:35203,127.28.104.253:46139,127.28.104.252:33477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:34:23.391486 3305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 FOLLOWER]: Refusing update from remote peer 573e48de9df3491b85efe17e27d6a911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 06:34:23.391261 3360 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d9102c0e7dfe4c00bae643db3ddc784f (127.28.104.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } } }
I20260619 06:34:23.397464 3361 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:23.401160 3360 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.104.253:46139
I20260619 06:34:23.401945 3263 master.cc:537] Master@127.28.104.253:46139 shutting down...
I20260619 06:34:23.401903 3361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } } }
I20260619 06:34:23.402791 3361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:23.417732 3305 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d9102c0e7dfe4c00bae643db3ddc784f (127.28.104.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } } }
I20260619 06:34:23.434393 3337 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 573e48de9df3491b85efe17e27d6a911. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } } }
I20260619 06:34:23.434696 3337 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: LEARNER
I20260619 06:34:23.435097 3367 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } } }
I20260619 06:34:23.435397 3367 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: LEARNER
I20260619 06:34:23.435397 3371 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } } }
I20260619 06:34:23.435806 3371 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: LEARNER
I20260619 06:34:23.437525 3263 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 LEARNER]: Raft consensus shutting down.
I20260619 06:34:23.438094 3263 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 LEARNER]: Raft consensus is shut down!
I20260619 06:34:23.438266 3263 tablet_replica.cc:330] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: stopping tablet replica
I20260619 06:34:23.447384 3263 master.cc:552] Master@127.28.104.253:46139 shutdown complete.
I20260619 06:34:23.447527 3263 master_runner.cc:305] Clearing existing system tablet
I20260619 06:34:23.448019 3263 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:23.448165 3263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:23.451742 3263 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:34:23.453655 3373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:23.454021 3263 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:23.454497 3263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:34:23.454641 3263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal
uuid: "d9102c0e7dfe4c00bae643db3ddc784f"
format_stamp: "Formatted at 2026-06-19 06:34:23 on dist-test-slave-nf4n"
server_key: "da1fc4809ba4c14e27394c23e083d23e"
server_key_iv: "e67b2d010e3f7bf528db15619e0be0e4"
server_key_version: "encryptionkey@0"
I20260619 06:34:23.459728 3263 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:23.468116 3263 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:34:23.468432 3263 log.cc:1196] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 06:34:23.469127 3263 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: Deleting consensus metadata
I20260619 06:34:23.469616 3263 master_runner.cc:315] Copying system tablet from 127.28.104.254:35203
I20260619 06:34:23.472839 3263 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: tablet copy: Beginning tablet copy session from remote peer at address 127.28.104.254:35203
I20260619 06:34:23.490572 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.473857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:53132)
0619 06:34:23.474555 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:23.474580 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:23.474884 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:23.475681 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:23.475688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:23.476300 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:23.476582 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.476594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.478278 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.478291 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.479915 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.479931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.480991 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.480997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.481218 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.481996 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:23.482020 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:23.483159 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:23.487437 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.487444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.487449 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.487974 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.488419 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.488424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.488428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.488610 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.489389 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:23.489395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:23.489574 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 06:34:23.489748 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 06:34:23.490085 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":350,"threads_started":1}
I20260619 06:34:23.491581 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.474001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.28.104.254:35203)
0619 06:34:23.474228 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.474237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.474707 (+ 470us) server_negotiation.cc:407] Connection header received
0619 06:34:23.475119 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.475129 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.475239 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.475367 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.476794 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.478114 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.480090 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.480758 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.480912 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.483387 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.483394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.483400 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.483443 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.487272 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.488110 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.488116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.488120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.488294 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.488736 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.488741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.488744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.489248 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.489421 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.490976 (+ 1555us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.491204 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:34:23.492507 2817 tablet_copy_service.cc:140] P 573e48de9df3491b85efe17e27d6a911: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d9102c0e7dfe4c00bae643db3ddc784f ({username='oryx', principal='oryx/127.28.104.253@KRBTEST.COM'} at 127.0.0.1:53132)
I20260619 06:34:23.492815 2817 tablet_copy_service.cc:161] P 573e48de9df3491b85efe17e27d6a911: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d9102c0e7dfe4c00bae643db3ddc784f at {username='oryx', principal='oryx/127.28.104.253@KRBTEST.COM'} at 127.0.0.1:53132: session id = d9102c0e7dfe4c00bae643db3ddc784f-00000000000000000000000000000000
I20260619 06:34:23.497435 2817 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:34:23.499769 3263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:34:23.507419 3263 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: tablet copy: Starting download of 0 data blocks...
I20260619 06:34:23.507771 3263 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: tablet copy: Starting download of 1 WAL segments...
I20260619 06:34:23.511049 3263 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:34:23.515767 2817 tablet_copy_service.cc:342] P 573e48de9df3491b85efe17e27d6a911: Request end of tablet copy session d9102c0e7dfe4c00bae643db3ddc784f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.104.253@KRBTEST.COM'} at 127.0.0.1:53132
I20260619 06:34:23.516008 2817 tablet_copy_service.cc:433] P 573e48de9df3491b85efe17e27d6a911: ending tablet copy session d9102c0e7dfe4c00bae643db3ddc784f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d9102c0e7dfe4c00bae643db3ddc784f
I20260619 06:34:23.526798 3263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:23.529491 3381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:23.539043 3382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:23.539740 3384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:23.540190 3263 server_base.cc:1034] running on GCE node
I20260619 06:34:23.540534 3263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:23.541236 3263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:23.542423 3263 hybrid_clock.cc:648] HybridClock initialized: now 1781850863542338 us; error 59 us; skew 500 ppm
I20260619 06:34:23.542685 3263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850863, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:23.544646 3263 init.cc:377] Logged in from keytab as oryx/127.28.104.253@KRBTEST.COM (short username oryx)
I20260619 06:34:23.545934 3263 webserver.cc:460] Webserver started at http://127.28.104.253:36351/ using document root <none> and password file <none>
I20260619 06:34:23.546265 3263 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:23.546355 3263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:23.548687 3263 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:34:23.550588 3391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:23.551115 3263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:34:23.551257 3263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal
uuid: "d9102c0e7dfe4c00bae643db3ddc784f"
format_stamp: "Formatted at 2026-06-19 06:34:23 on dist-test-slave-nf4n"
server_key: "da1fc4809ba4c14e27394c23e083d23e"
server_key_iv: "e67b2d010e3f7bf528db15619e0be0e4"
server_key_version: "encryptionkey@0"
I20260619 06:34:23.551436 3263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:23.562466 3263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:23.566067 3263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:23.591871 3263 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.253:46139
I20260619 06:34:23.591903 3442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.253:46139 every 8 connection(s)
I20260619 06:34:23.592831 3263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
I20260619 06:34:23.599726 3443 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:34:23.605171 3443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: Bootstrap starting.
I20260619 06:34:23.615873 3443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:34:23.616367 3443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: Bootstrap complete.
I20260619 06:34:23.617277 3443 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } }
I20260619 06:34:23.617506 3443 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d9102c0e7dfe4c00bae643db3ddc784f, State: Initialized, Role: LEARNER
I20260619 06:34:23.617780 3443 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } }
I20260619 06:34:23.618574 3446 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } } }
I20260619 06:34:23.618899 3443 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:23.618935 3446 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: LEARNER
I20260619 06:34:23.630224 3457 catalog_manager.cc:1259] Loaded cluster ID: 7951725ab48a4ccfae5b607186e2a0c7
I20260619 06:34:23.630410 3457 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: loading cluster ID for follower catalog manager: success
I20260619 06:34:23.633381 3457 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: acquiring CA information for follower catalog manager: success
I20260619 06:34:23.635214 3457 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:34:23.747207 3258 heartbeater.cc:503] Master 127.28.104.254:35203 was elected leader, sending a full tablet report...
W20260619 06:34:23.818382 3362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:23.818769 3362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:23.818934 3362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:23.826797 3362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:23.826897 3362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:23.826967 3362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:23.827052 3362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:23.827133 3362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:23.837395 3362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:35203,127.28.104.253:46139,127.28.104.252:33477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.252.keytab
--principal=oryx/127.28.104.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.252:33477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.104.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:23.838771 3362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:23.840131 3362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:23.852598 3465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:23.852532 3462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:23.852636 3463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:23.853739 3362 server_base.cc:1034] running on GCE node
I20260619 06:34:23.854552 3362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:23.856117 3362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:23.857409 3362 hybrid_clock.cc:648] HybridClock initialized: now 1781850863857280 us; error 56 us; skew 500 ppm
I20260619 06:34:23.857887 3362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850863, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:23.861749 3362 init.cc:377] Logged in from keytab as oryx/127.28.104.252@KRBTEST.COM (short username oryx)
I20260619 06:34:23.863276 3362 webserver.cc:460] Webserver started at http://127.28.104.252:39173/ using document root <none> and password file <none>
I20260619 06:34:23.864009 3362 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:23.864171 3362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:23.864495 3362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:23.867758 3362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data/instance:
uuid: "aa378990b19c4102b7d801f2c71a7a41"
format_stamp: "Formatted at 2026-06-19 06:34:23 on dist-test-slave-nf4n"
server_key: "682d6384e77b7d9ec7a1347091055729"
server_key_iv: "4768977d80de23898f6fb97955b9ac1a"
server_key_version: "encryptionkey@0"
I20260619 06:34:23.868564 3362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal/instance:
uuid: "aa378990b19c4102b7d801f2c71a7a41"
format_stamp: "Formatted at 2026-06-19 06:34:23 on dist-test-slave-nf4n"
server_key: "682d6384e77b7d9ec7a1347091055729"
server_key_iv: "4768977d80de23898f6fb97955b9ac1a"
server_key_version: "encryptionkey@0"
I20260619 06:34:23.873852 3362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 06:34:23.877360 3472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:23.878871 3362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:34:23.879057 3362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal
uuid: "aa378990b19c4102b7d801f2c71a7a41"
format_stamp: "Formatted at 2026-06-19 06:34:23 on dist-test-slave-nf4n"
server_key: "682d6384e77b7d9ec7a1347091055729"
server_key_iv: "4768977d80de23898f6fb97955b9ac1a"
server_key_version: "encryptionkey@0"
I20260619 06:34:23.879259 3362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:23.893841 3362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:23.898355 3362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:23.898666 3362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:23.917136 3362 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.252:33477
I20260619 06:34:23.917150 3523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.252:33477 every 8 connection(s)
I20260619 06:34:23.918716 3362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
I20260619 06:34:23.922873 3525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:23.928043 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 3362
I20260619 06:34:23.929867 3525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850863, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.252@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.252@KRBTEST.COM
I20260619 06:34:23.949476 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.932675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.28.104.254:35203)
0619 06:34:23.932887 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.932895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.933743 (+ 848us) server_negotiation.cc:407] Connection header received
0619 06:34:23.934822 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.934827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.934892 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.935009 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.936379 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.937153 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.938725 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.939094 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.939239 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.943175 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.943184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.943192 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.943244 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.946212 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.946990 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.946995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.946999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.947132 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.947551 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.947557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.947561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.947856 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.948013 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.948879 (+ 866us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.949142 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:34:23.950459 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.932671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:53138)
0619 06:34:23.933587 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:23.933630 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:23.933862 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:23.935079 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:23.935088 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:23.935773 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:23.936208 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.936226 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.937326 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.937331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.938579 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.938590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.939341 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.939349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.939537 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.940222 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:23.940243 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:23.942956 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:23.946349 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.946362 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.946375 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.946858 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.947240 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.947246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.947250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.947441 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.947984 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:23.947991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:23.948443 (+ 452us) client_negotiation.cc:769] Sending connection context
0619 06:34:23.948761 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 06:34:23.949359 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":34,"spinlock_wait_cycles":14720,"thread_start_us":530,"threads_started":1}
I20260619 06:34:23.951179 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.930352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.28.104.252:33477)
0619 06:34:23.931223 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.931242 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.931285 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:34:23.933422 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.933452 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.933903 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.934163 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.935649 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.937189 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.938452 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.939063 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.939229 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.943838 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.943846 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.943859 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.943887 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.946634 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.947320 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.947335 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.947337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.947478 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.947851 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.947857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.947860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.948411 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.948706 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.949131 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.950878 (+ 1747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1573,"server-negotiator.queue_time_us":590,"thread_start_us":364,"threads_started":1}
I20260619 06:34:23.952255 3525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850863, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.252@KRBTEST.COM for oryx/127.28.104.253@KRBTEST.COM
I20260619 06:34:23.965277 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.952977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:46139 (local address 127.0.0.1:43540)
0619 06:34:23.953166 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:23.953187 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:23.953297 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:23.954031 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:23.954035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:23.954368 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:23.954498 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.954505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.955730 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.955733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.956709 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.956720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.957482 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.957486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.957607 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.958127 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:23.958142 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:23.960028 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:23.963250 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.963254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.963256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.963647 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.963954 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.963957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.963959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.964079 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.964518 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:23.964521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:23.964646 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 06:34:23.964753 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 06:34:23.964957 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 06:34:23.965301 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.953129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.28.104.253:46139)
0619 06:34:23.953706 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.953712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.953730 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:34:23.953790 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.953795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.953876 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.953992 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.954631 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.955608 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.956833 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.957349 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.957491 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.960245 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.960254 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.960261 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.960299 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.963130 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.963744 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.963747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.963749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.963862 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.964164 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.964167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.964169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.964424 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.964537 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.964834 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.964989 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":302,"threads_started":1}
I20260619 06:34:23.966375 3525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:34:23 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850863, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.252@KRBTEST.COM for oryx/127.28.104.252@KRBTEST.COM
I20260619 06:34:23.974181 3531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } } }
I20260619 06:34:23.974560 3531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:23.982056 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.967217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:33477 (local address 127.0.0.1:46892)
0619 06:34:23.967345 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:23.967360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:23.967468 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:23.967730 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:23.967733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:23.968171 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:23.968331 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.968339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.969312 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.969315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.970138 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.970147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.970754 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.970758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.970954 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.971496 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:23.971512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:23.973637 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:23.978518 (+ 4881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.978525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.978532 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.979182 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.979675 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:23.979682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:23.979688 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:23.979896 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:23.980722 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:23.980729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:23.980903 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 06:34:23.981116 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 06:34:23.981492 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:34:23.982884 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.967164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.28.104.252:33477)
0619 06:34:23.967327 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.967333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.967421 (+ 88us) server_negotiation.cc:407] Connection header received
0619 06:34:23.967553 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.967557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.967608 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.967701 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:23.968451 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.969203 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.970253 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.970631 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.970728 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.973848 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:23.973855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:23.973859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:23.973895 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:23.978289 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.979336 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.979344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.979350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.979522 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:23.980028 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:23.980035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:23.980040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:23.980497 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:23.980667 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.982505 (+ 1838us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.982697 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:34:23.987826 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.974718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.28.104.253:46139)
0619 06:34:23.979993 (+ 5275us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:23.980003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:23.981850 (+ 1847us) server_negotiation.cc:407] Connection header received
0619 06:34:23.982075 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:23.982084 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:23.982216 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:23.982855 (+ 639us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:23.983030 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.983811 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.986154 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.986979 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.987114 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.987262 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:23.987554 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:34:23.987657 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5075}
I20260619 06:34:23.987975 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:23.974287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:46139 (local address 127.0.0.1:43546)
0619 06:34:23.981737 (+ 7450us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:23.981754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:23.981931 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:23.982423 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:23.982430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:23.982467 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:23.982725 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.982736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.983964 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.983970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.985984 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:23.986001 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:23.987171 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:23.987178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:23.987360 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:23.987420 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 06:34:23.987526 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:34:23.987662 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7323,"thread_start_us":7212,"threads_started":1}
I20260619 06:34:23.988767 3525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Bootstrap starting.
I20260619 06:34:23.988986 3446 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 573e48de9df3491b85efe17e27d6a911. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: NON_VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: true } } }
I20260619 06:34:23.989336 3446 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: LEARNER
I20260619 06:34:23.989871 3532 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911: attempting to promote NON_VOTER d9102c0e7dfe4c00bae643db3ddc784f to VOTER
I20260619 06:34:23.990841 3532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } }
I20260619 06:34:23.992782 3418 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 LEARNER]: Refusing update from remote peer 573e48de9df3491b85efe17e27d6a911: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 06:34:23.992823 3525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:23.993392 3533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 06:34:23.994140 3525 log.cc:824] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:23.996300 3532 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d9102c0e7dfe4c00bae643db3ddc784f (127.28.104.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } }
I20260619 06:34:23.996860 3418 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d9102c0e7dfe4c00bae643db3ddc784f (127.28.104.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } }
I20260619 06:34:23.997231 3525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: No bootstrap required, opened a new log
I20260619 06:34:23.999481 3446 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } }
I20260619 06:34:23.999711 3446 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:34:24.000588 3533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } }
I20260619 06:34:24.000618 3532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } }
I20260619 06:34:24.000829 3533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:24.000851 3532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:24.005864 3525 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } }
I20260619 06:34:24.006254 3525 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:24.006390 3525 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa378990b19c4102b7d801f2c71a7a41, State: Initialized, Role: FOLLOWER
I20260619 06:34:24.006984 3525 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } }
I20260619 06:34:24.007817 3538 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } } }
I20260619 06:34:24.008170 3538 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:34:24.008919 3525 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:34:24.017868 3549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:34:24.018024 3549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:34:24.022074 3362 master_runner.cc:428] Detected that this master aa378990b19c4102b7d801f2c71a7a41 is joining an existing cluster
I20260619 06:34:24.022228 3362 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.104.252:33477
I20260619 06:34:24.045115 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.026254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.28.104.254:35203)
0619 06:34:24.026447 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.026453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.028123 (+ 1670us) server_negotiation.cc:407] Connection header received
0619 06:34:24.028326 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.028331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.028408 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.028512 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:24.029927 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.031471 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.034298 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.034971 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.035065 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.037854 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:24.037861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:24.037867 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:24.037918 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:24.041747 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.042556 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.042563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.042567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.042728 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.043227 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.043232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.043236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.043604 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:24.043750 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.044491 (+ 741us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.044726 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:34:24.047618 3555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.026713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:53142)
0619 06:34:24.028017 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:24.028042 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:24.028183 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:24.028533 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:24.028538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:24.029210 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:24.029698 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.029715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.031728 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.031736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.034122 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.034135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.035346 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.035352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.035541 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.036288 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:24.036322 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:24.037642 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:24.041895 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.041900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.041903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.042370 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.042886 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.042893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.042897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.043095 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.043841 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:24.043848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:24.044099 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 06:34:24.046766 (+ 2667us) client_negotiation.cc:240] Negotiation successful
0619 06:34:24.047122 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"thread_start_us":6059,"threads_started":1}
I20260619 06:34:24.049008 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.026332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.28.104.253:46139)
0619 06:34:24.026552 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.026560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.032632 (+ 6072us) server_negotiation.cc:407] Connection header received
0619 06:34:24.032827 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.032832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.032915 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.033039 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:24.034794 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.035911 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.038245 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.039012 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.039113 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.042081 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:24.042087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:24.042092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:24.042132 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:24.045693 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.046555 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.046560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.046563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.046712 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.047195 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.047200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.047203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.047536 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:24.047683 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.048377 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.048603 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:34:24.049448 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.027055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.28.104.252:33477)
0619 06:34:24.027243 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.027251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.032978 (+ 5727us) server_negotiation.cc:407] Connection header received
0619 06:34:24.033227 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.033232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.033301 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.033374 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:24.034852 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.035972 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.038226 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.038937 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.039028 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.042121 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:24.042127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:24.042131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:24.042164 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:24.046347 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.047136 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.047141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.047144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.047311 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.047706 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.047711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.047714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.048091 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:24.048235 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.048724 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.048927 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":56}
I20260619 06:34:24.050105 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.026766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:46139 (local address 127.0.0.1:43554)
0619 06:34:24.032530 (+ 5764us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:24.032553 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:24.032693 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:24.033123 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:24.033127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:24.034219 (+ 1092us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:24.034608 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.034623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.036120 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.036128 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.038075 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.038093 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.039426 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.039433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.039624 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.040428 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:24.040456 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:24.041879 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:24.045851 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.045866 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.045869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.046414 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.046850 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.046854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.046858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.047057 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.047794 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:24.047801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:24.048079 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 06:34:24.049228 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0619 06:34:24.049572 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5604,"mutex_wait_us":102,"spinlock_wait_cycles":1280,"thread_start_us":940,"threads_started":1}
I20260619 06:34:24.050716 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.027122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:33477 (local address 127.0.0.1:46896)
0619 06:34:24.032878 (+ 5756us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:24.032901 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:24.033100 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:24.033605 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:24.033610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:24.034285 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:24.034706 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.034718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.036189 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.036194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.038083 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.038096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.039320 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.039326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.039557 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.040571 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:24.040626 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:24.041957 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:24.046504 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.046509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.046513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.047007 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.047453 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.047458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.047461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.047597 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.048302 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:24.048306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:24.048449 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 06:34:24.049676 (+ 1227us) client_negotiation.cc:240] Negotiation successful
0619 06:34:24.049962 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5626,"mutex_wait_us":92,"thread_start_us":5075,"threads_started":1}
I20260619 06:34:24.057195 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 2858
Jun 19 06:34:24 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850861, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.254@KRBTEST.COM for oryx/127.28.104.252@KRBTEST.COM
I20260619 06:34:24.069772 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.054119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:33477 (local address 127.0.0.1:46904)
0619 06:34:24.054339 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:24.054358 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:24.054486 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:24.055069 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:24.055072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:24.055443 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:24.055577 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.055587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.057012 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.057040 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.058287 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.058301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.059244 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.059250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.059435 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.060120 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:24.060147 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:24.062844 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:24.066899 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.066905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.066909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.067476 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.067919 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.067925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.067931 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.068101 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.068752 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:24.068758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:24.068947 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 06:34:24.069137 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 06:34:24.069435 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
Jun 19 06:34:24 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.193@KRBTEST.COM
I20260619 06:34:24.069850 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.054166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.28.104.252:33477)
0619 06:34:24.054702 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.054707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.054723 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:34:24.054820 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.054826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.054930 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.055026 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:24.055722 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.056851 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.058452 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.059066 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.061278 (+ 2212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.063038 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:24.063044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:24.063049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:24.063086 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:24.066749 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.067612 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.067619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.067624 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.067785 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.068223 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.068229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.068234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.068619 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:24.068766 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.069262 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.069491 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416}
I20260619 06:34:24.071385 2797 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.104.252:33477: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41"
member_type: NON_VOTER
last_known_addr {
host: "127.28.104.252"
port: 33477
}
attrs {
promote: true
}
}
dest_uuid: "573e48de9df3491b85efe17e27d6a911"
cas_config_opid_index: 6
I20260619 06:34:24.072045 2797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } }
I20260619 06:34:24.074095 3418 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 FOLLOWER]: Refusing update from remote peer 573e48de9df3491b85efe17e27d6a911: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 06:34:24.074393 3499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:24.074751 3533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 06:34:24.077230 3532 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa378990b19c4102b7d801f2c71a7a41 (127.28.104.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } } }
I20260619 06:34:24.077762 3418 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER aa378990b19c4102b7d801f2c71a7a41 (127.28.104.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } } }
I20260619 06:34:24.078258 3499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 FOLLOWER]: Refusing update from remote peer 573e48de9df3491b85efe17e27d6a911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 06:34:24.080806 3532 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.104.252:33477
I20260619 06:34:24.080864 3446 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } } }
I20260619 06:34:24.081228 3446 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:34:24.081193 3531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } } }
I20260619 06:34:24.081427 3531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:24.081499 3362 master.cc:537] Master@127.28.104.252:33477 shutting down...
I20260619 06:34:24.082283 3533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:24.085258 3532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } } }
I20260619 06:34:24.085707 3532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
W20260619 06:34:24.085804 2781 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 06:34:24.096745 3558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.058578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.28.104.193:37853)
0619 06:34:24.059236 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.059246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.059276 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:34:24.059351 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.059359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.059664 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.059832 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:24.061460 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.062604 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.064353 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.065336 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.065486 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.086760 (+ 21274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:24.086782 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:24.086802 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:24.086853 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:24.091274 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.092369 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.092380 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.092385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.092549 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.093101 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.093108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.093112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.094430 (+ 1318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:24.095881 (+ 1451us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.096039 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.096333 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":266,"threads_started":1}
I20260619 06:34:24.098462 2983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:24.104180 3362 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:24.104664 3362 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:24.104919 3362 tablet_replica.cc:330] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: stopping tablet replica
W20260619 06:34:24.109534 2781 consensus_peers.cc:487] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 -> Peer aa378990b19c4102b7d801f2c71a7a41 (127.28.104.252:33477): Couldn't send request to peer aa378990b19c4102b7d801f2c71a7a41. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 06:34:24.113221 3362 master.cc:552] Master@127.28.104.252:33477 shutdown complete.
I20260619 06:34:24.113308 3362 master_runner.cc:305] Clearing existing system tablet
I20260619 06:34:24.113683 3362 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:24.113794 3362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:24.122138 3362 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.006s
I20260619 06:34:24.123889 3562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:24.124218 3362 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:24.124502 3362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:34:24.124604 3362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal
uuid: "aa378990b19c4102b7d801f2c71a7a41"
format_stamp: "Formatted at 2026-06-19 06:34:23 on dist-test-slave-nf4n"
server_key: "682d6384e77b7d9ec7a1347091055729"
server_key_iv: "4768977d80de23898f6fb97955b9ac1a"
server_key_version: "encryptionkey@0"
I20260619 06:34:24.128160 3362 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:24.132961 3362 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:34:24.133224 3362 log.cc:1196] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 06:34:24.133651 3362 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Deleting consensus metadata
I20260619 06:34:24.133885 3362 master_runner.cc:315] Copying system tablet from 127.28.104.254:35203
I20260619 06:34:24.138616 3362 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: tablet copy: Beginning tablet copy session from remote peer at address 127.28.104.254:35203
I20260619 06:34:24.153329 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.139407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.28.104.254:35203)
0619 06:34:24.139670 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.139680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.141530 (+ 1850us) server_negotiation.cc:407] Connection header received
0619 06:34:24.141779 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.141785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.141862 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.141998 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:24.142810 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.143610 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.144883 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.145374 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.145511 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.147340 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:24.147347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:24.147351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:24.147390 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:24.150508 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.151120 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.151126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.151129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.151300 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.151764 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.151769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.151772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.152054 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:24.152220 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.152699 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.152928 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 06:34:24.153415 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.139351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:53154)
0619 06:34:24.141414 (+ 2063us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:24.141430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:24.141639 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:24.141997 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:24.142002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:24.142446 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:24.142631 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.142639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.143753 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.143759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.144730 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.144742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.145535 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.145540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.145725 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.146361 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:24.146379 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:24.147148 (+ 769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:24.150634 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.150640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.150644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.150997 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.151429 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.151433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.151437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.151589 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.152172 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:24.152177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:24.152458 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 06:34:24.152606 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 06:34:24.152962 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1924,"thread_start_us":1839,"threads_started":1}
I20260619 06:34:24.154248 2817 tablet_copy_service.cc:140] P 573e48de9df3491b85efe17e27d6a911: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer aa378990b19c4102b7d801f2c71a7a41 ({username='oryx', principal='oryx/127.28.104.252@KRBTEST.COM'} at 127.0.0.1:53154)
I20260619 06:34:24.154426 2817 tablet_copy_service.cc:161] P 573e48de9df3491b85efe17e27d6a911: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer aa378990b19c4102b7d801f2c71a7a41 at {username='oryx', principal='oryx/127.28.104.252@KRBTEST.COM'} at 127.0.0.1:53154: session id = aa378990b19c4102b7d801f2c71a7a41-00000000000000000000000000000000
I20260619 06:34:24.156674 2817 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:34:24.158322 3362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:34:24.163223 3362 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: tablet copy: Starting download of 0 data blocks...
I20260619 06:34:24.163518 3362 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: tablet copy: Starting download of 1 WAL segments...
I20260619 06:34:24.165724 3362 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:34:24.169183 2817 tablet_copy_service.cc:342] P 573e48de9df3491b85efe17e27d6a911: Request end of tablet copy session aa378990b19c4102b7d801f2c71a7a41-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.104.252@KRBTEST.COM'} at 127.0.0.1:53154
I20260619 06:34:24.169327 2817 tablet_copy_service.cc:433] P 573e48de9df3491b85efe17e27d6a911: ending tablet copy session aa378990b19c4102b7d801f2c71a7a41-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer aa378990b19c4102b7d801f2c71a7a41
I20260619 06:34:24.179044 3362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:24.181778 3569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:24.186657 3570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:24.194456 3572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:24.200874 3362 server_base.cc:1034] running on GCE node
I20260619 06:34:24.201251 3362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:24.202003 3362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:24.203264 3362 hybrid_clock.cc:648] HybridClock initialized: now 1781850864203210 us; error 46 us; skew 500 ppm
I20260619 06:34:24.203521 3362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:24 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850864, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:24.205420 3362 init.cc:377] Logged in from keytab as oryx/127.28.104.252@KRBTEST.COM (short username oryx)
I20260619 06:34:24.206754 3362 webserver.cc:460] Webserver started at http://127.28.104.252:40089/ using document root <none> and password file <none>
I20260619 06:34:24.207062 3362 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:24.207172 3362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:24.209585 3362 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:34:24.211237 3579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:24.211637 3362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:34:24.211761 3362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal
uuid: "aa378990b19c4102b7d801f2c71a7a41"
format_stamp: "Formatted at 2026-06-19 06:34:23 on dist-test-slave-nf4n"
server_key: "682d6384e77b7d9ec7a1347091055729"
server_key_iv: "4768977d80de23898f6fb97955b9ac1a"
server_key_version: "encryptionkey@0"
I20260619 06:34:24.211958 3362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:24.229609 3362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:24.233395 3362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:24.267431 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 2858
I20260619 06:34:24.271181 3362 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.252:33477
I20260619 06:34:24.271231 3630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.252:33477 every 8 connection(s)
I20260619 06:34:24.272495 3362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
I20260619 06:34:24.277735 3631 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:34:24.281831 3631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Bootstrap starting.
I20260619 06:34:24.293447 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:37853
--local_ip_for_outbound_sockets=127.28.104.193
--tserver_master_addrs=127.28.104.254:35203
--webserver_port=35623
--webserver_interface=127.28.104.193
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab
--principal=oryx/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.104.254:35203,127.28.104.253:46139
--tserver_master_addrs=127.28.104.254:35203,127.28.104.253:46139,127.28.104.252:33477 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:34:24.301136 3631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:34:24.301640 3631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Bootstrap complete.
I20260619 06:34:24.302739 3631 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } }
I20260619 06:34:24.303040 3631 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa378990b19c4102b7d801f2c71a7a41, State: Initialized, Role: LEARNER
I20260619 06:34:24.303290 3631 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } }
I20260619 06:34:24.304122 3635 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } } }
I20260619 06:34:24.304432 3631 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:24.304593 3635 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:34:24.321691 3646 catalog_manager.cc:1259] Loaded cluster ID: 7951725ab48a4ccfae5b607186e2a0c7
I20260619 06:34:24.321802 3646 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: loading cluster ID for follower catalog manager: success
I20260619 06:34:24.325567 3646 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: acquiring CA information for follower catalog manager: success
I20260619 06:34:24.328135 3646 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:34:24.519842 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.513825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:33477 (local address 127.0.0.1:43842)
0619 06:34:24.514051 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:24.514073 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:24.514245 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:24.515062 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:24.515066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:24.515107 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:24.515318 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.515329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.516601 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.516607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.517801 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.517813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.519011 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.519019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.519213 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.519246 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:34:24.519368 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 06:34:24.519522 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 06:34:24.519871 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.514087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.28.104.252:33477)
0619 06:34:24.514746 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.514750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.514768 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:34:24.514835 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.514838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.514904 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.515041 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:24.515488 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.516446 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.517964 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.518837 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.519047 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.519287 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.519389 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.519522 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":393,"threads_started":1}
I20260619 06:34:24.521322 3606 raft_consensus.cc:1203] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260619 06:34:24.522223 3635 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 573e48de9df3491b85efe17e27d6a911. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } } }
I20260619 06:34:24.522567 3635 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: This master's current role is: LEARNER
W20260619 06:34:24.643671 3634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:24.643965 3634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:24.644057 3634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:24.651346 3634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:24.651496 3634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:24.651687 3634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:34:24.661580 3634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab
--principal=oryx/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:37853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=35623
--tserver_master_addrs=127.28.104.254:35203,127.28.104.253:46139,127.28.104.252:33477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:24.662837 3634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:24.664105 3634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:24.677474 3655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:24.677392 3653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:24.677553 3652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:24.678364 3634 server_base.cc:1034] running on GCE node
I20260619 06:34:24.678962 3634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:24.679852 3634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:24.681118 3634 hybrid_clock.cc:648] HybridClock initialized: now 1781850864681065 us; error 64 us; skew 500 ppm
I20260619 06:34:24.681434 3634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:24 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850864, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:24.684963 3634 init.cc:377] Logged in from keytab as oryx/127.28.104.193@KRBTEST.COM (short username oryx)
I20260619 06:34:24.686380 3634 webserver.cc:460] Webserver started at http://127.28.104.193:35623/ using document root <none> and password file <none>
I20260619 06:34:24.687078 3634 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:24.687237 3634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:24.692478 3634 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:34:24.695086 3662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:24.696350 3634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:34:24.696552 3634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "4fad813d2f6144bda0cba9417d0af8b8"
format_stamp: "Formatted at 2026-06-19 06:34:21 on dist-test-slave-nf4n"
server_key: "41e5c8460cfd4ec4a78b41a8c5dab719"
server_key_iv: "426ae9a3a5a70f5a73a351f5696e8073"
server_key_version: "encryptionkey@0"
I20260619 06:34:24.697613 3634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:24.715680 3634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:24.720058 3634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:24.720342 3634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:24.721450 3634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:24.722942 3634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:24.723080 3634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:24.723202 3634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:24.723286 3634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:24.757007 3634 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:37853
I20260619 06:34:24.757162 3776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:37853 every 8 connection(s)
I20260619 06:34:24.758662 3634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:34:24.767527 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 3634
I20260619 06:34:24.767918 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 2995
Jun 19 06:34:24 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.194@KRBTEST.COM
Jun 19 06:34:24 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850864, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.253@KRBTEST.COM
Jun 19 06:34:24 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850864, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.252@KRBTEST.COM
Jun 19 06:34:24 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850864, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:24.812536 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.762872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:33711 (local address 127.28.104.252:33477)
0619 06:34:24.763084 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.763093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.787461 (+ 24368us) server_negotiation.cc:407] Connection header received
0619 06:34:24.791644 (+ 4183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.791650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.791738 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.791869 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:24.793853 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.794897 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.797949 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.798717 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.798854 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.804824 (+ 5970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:24.804834 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:24.804840 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:24.804893 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:24.808589 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.810301 (+ 1712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.810308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.810313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.810466 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.810898 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.810903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.810906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.811217 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:24.811313 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.811949 (+ 636us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.812199 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:34:24.812580 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.761899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:43109 (local address 127.28.104.254:35203)
0619 06:34:24.787277 (+ 25378us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.787289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.791119 (+ 3830us) server_negotiation.cc:407] Connection header received
0619 06:34:24.793376 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.793386 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.793485 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.793604 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:24.794903 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.796266 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.798579 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.799260 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.799356 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.805371 (+ 6015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:24.805379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:24.805385 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:24.805429 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:24.809408 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.810477 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.810483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.810485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.810622 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.811080 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.811085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.811088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.811407 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:24.811560 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.811924 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.812204 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25168,"thread_start_us":25428,"threads_started":1}
I20260619 06:34:24.812584 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.761829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:57989 (local address 127.28.104.253:46139)
0619 06:34:24.787816 (+ 25987us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.787820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.789788 (+ 1968us) server_negotiation.cc:407] Connection header received
0619 06:34:24.791680 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.791687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.791790 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.791881 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:24.793856 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.794955 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.797927 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.798610 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.798714 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.804308 (+ 5594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:24.804314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:24.804318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:24.804374 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:24.808209 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.809315 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.809323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.809327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.809511 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.810289 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.810297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.810302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.810750 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:24.810889 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.811941 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.812181 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25868,"thread_start_us":25759,"threads_started":1}
I20260619 06:34:24.814002 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.762379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:33477 (local address 127.28.104.193:33711)
0619 06:34:24.787177 (+ 24798us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:24.787277 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:24.791365 (+ 4088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:24.792119 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:24.792128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:24.792709 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:24.793647 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.793674 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.795088 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.795093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.797735 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.797751 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.799188 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.799196 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.799422 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.800381 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:24.800410 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:24.804552 (+ 4142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:24.809521 (+ 4969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.809527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.809534 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.810157 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.810585 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.810588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.810590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.810746 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.811374 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:24.811377 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:24.811548 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 06:34:24.812853 (+ 1305us) client_negotiation.cc:240] Negotiation successful
0619 06:34:24.813188 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24329,"mutex_wait_us":52,"spinlock_wait_cycles":11520,"thread_start_us":27394,"threads_started":1}
I20260619 06:34:24.813963 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.762379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:46139 (local address 127.28.104.193:57989)
0619 06:34:24.789645 (+ 27266us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:24.789670 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:24.791394 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:24.792181 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:24.792186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:24.792856 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:24.793648 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.793678 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.795205 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.795210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.797741 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.797756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.799055 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.799063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.799411 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.800193 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:24.800256 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:24.803953 (+ 3697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:24.808394 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.808404 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.808424 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.809166 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.809876 (+ 710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.809881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.809885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.810155 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.810982 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:24.810990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:24.811438 (+ 448us) client_negotiation.cc:769] Sending connection context
0619 06:34:24.813054 (+ 1616us) client_negotiation.cc:240] Negotiation successful
0619 06:34:24.813339 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27080,"mutex_wait_us":357,"spinlock_wait_cycles":67712,"thread_start_us":27759,"threads_started":1}
I20260619 06:34:24.813935 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.763232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.193:43109)
0619 06:34:24.790975 (+ 27743us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:24.790995 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:24.793165 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:24.793839 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:24.793844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:24.794397 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:24.794692 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.794704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.796474 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.796480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.798415 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:24.798428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.799651 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.799660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:24.799892 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.800554 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:24.800578 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:24.805105 (+ 4527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:24.809681 (+ 4576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.809688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.809695 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.810364 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.810741 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:24.810746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:24.810750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:24.810976 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:24.811536 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:24.811540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:24.811655 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 06:34:24.811820 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 06:34:24.812366 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27592,"mutex_wait_us":37,"thread_start_us":24292,"threads_started":1}
I20260619 06:34:24.815820 3779 heartbeater.cc:347] Connected to a master server at 127.28.104.253:46139
I20260619 06:34:24.815994 3778 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35203
I20260619 06:34:24.816084 3777 heartbeater.cc:347] Connected to a master server at 127.28.104.252:33477
I20260619 06:34:24.816249 3777 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:24.816246 3779 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:24.816246 3778 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:24.817400 3777 heartbeater.cc:511] Master 127.28.104.252:33477 requested a full tablet report, sending...
I20260619 06:34:24.817493 3778 heartbeater.cc:511] Master 127.28.104.254:35203 requested a full tablet report, sending...
I20260619 06:34:24.818274 3779 heartbeater.cc:511] Master 127.28.104.253:46139 requested a full tablet report, sending...
I20260619 06:34:24.818297 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:24.769215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.28.104.194:45547)
0619 06:34:24.788261 (+ 19046us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:24.788269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:24.788299 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:34:24.788368 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:24.788374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:24.788596 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:24.788807 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:24.791306 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.792967 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.794960 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:24.795874 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:24.796030 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:24.806187 (+ 10157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:24.806212 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:24.806232 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:24.806357 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:24.814109 (+ 7752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.815138 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.815147 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.815152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.815344 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:24.816343 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:24.816351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:24.816354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:24.816769 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:24.817124 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:24.817486 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 06:34:24.817760 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18891,"thread_start_us":18747,"threads_started":1}
I20260619 06:34:24.819617 3119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:24.820060 3596 ts_manager.cc:194] Registered new tserver with Master: 4fad813d2f6144bda0cba9417d0af8b8 (127.28.104.193:37853)
I20260619 06:34:24.820919 3408 ts_manager.cc:194] Registered new tserver with Master: 4fad813d2f6144bda0cba9417d0af8b8 (127.28.104.193:37853)
I20260619 06:34:24.821233 2797 ts_manager.cc:194] Re-registered known tserver with Master: 4fad813d2f6144bda0cba9417d0af8b8 (127.28.104.193:37853)
I20260619 06:34:24.822616 2797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:43109
I20260619 06:34:24.905543 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 2995
I20260619 06:34:24.922652 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:45547
--local_ip_for_outbound_sockets=127.28.104.194
--tserver_master_addrs=127.28.104.254:35203
--webserver_port=43217
--webserver_interface=127.28.104.194
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab
--principal=oryx/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.104.254:35203,127.28.104.253:46139
--tserver_master_addrs=127.28.104.254:35203,127.28.104.253:46139,127.28.104.252:33477 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:34:25.104784 3532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: NON_VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: true } } }
I20260619 06:34:25.105242 3532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:25.106765 3790 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911: attempting to promote NON_VOTER aa378990b19c4102b7d801f2c71a7a41 to VOTER
I20260619 06:34:25.107376 3790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } }
I20260619 06:34:25.109684 3418 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 FOLLOWER]: Refusing update from remote peer 573e48de9df3491b85efe17e27d6a911: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 06:34:25.109931 3606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 LEARNER]: Refusing update from remote peer 573e48de9df3491b85efe17e27d6a911: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 06:34:25.110301 3791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 06:34:25.110692 3532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 06:34:25.114405 3532 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa378990b19c4102b7d801f2c71a7a41 (127.28.104.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:25.114904 3418 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa378990b19c4102b7d801f2c71a7a41 (127.28.104.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:25.117547 3532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:25.117836 3532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:25.118054 3532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:25.118268 3532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:25.118453 3532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:25.118654 3532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:25.118115 3606 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, aa378990b19c4102b7d801f2c71a7a41 (127.28.104.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:25.125506 3797 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:25.125874 3797 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:34:25.126415 3796 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573e48de9df3491b85efe17e27d6a911" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:25.126816 3796 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: FOLLOWER
W20260619 06:34:25.295444 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:25.295800 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:25.295989 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:25.304353 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:25.304520 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:25.304705 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:34:25.317677 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab
--principal=oryx/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:45547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=43217
--tserver_master_addrs=127.28.104.254:35203,127.28.104.253:46139,127.28.104.252:33477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:25.319203 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:25.320729 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:25.336566 3802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:25.336472 3803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:25.336529 3805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:25.337469 3789 server_base.cc:1034] running on GCE node
I20260619 06:34:25.338207 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:25.339208 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:25.340426 3789 hybrid_clock.cc:648] HybridClock initialized: now 1781850865340359 us; error 42 us; skew 500 ppm
I20260619 06:34:25.340766 3789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:25 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850865, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:25.344184 3789 init.cc:377] Logged in from keytab as oryx/127.28.104.194@KRBTEST.COM (short username oryx)
I20260619 06:34:25.345762 3789 webserver.cc:460] Webserver started at http://127.28.104.194:43217/ using document root <none> and password file <none>
I20260619 06:34:25.346426 3789 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:25.346540 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:25.352149 3789 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:34:25.354995 3812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:25.356367 3789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:34:25.356567 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "c43f378aaf1940cc800b33e299cde2aa"
format_stamp: "Formatted at 2026-06-19 06:34:22 on dist-test-slave-nf4n"
server_key: "e0069f27255383834aa60c15e80f6629"
server_key_iv: "d78cfa4627c2bfa1d475dddca616f5e4"
server_key_version: "encryptionkey@0"
I20260619 06:34:25.357621 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:25.377118 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:25.381677 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:25.381958 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:25.382979 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:25.384477 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:25.384586 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:25.384712 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:25.384804 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:25.419257 3789 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:45547
I20260619 06:34:25.419369 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:45547 every 8 connection(s)
I20260619 06:34:25.420763 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:34:25.425658 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 3789
I20260619 06:34:25.426172 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 3129
Jun 19 06:34:25 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.195@KRBTEST.COM
Jun 19 06:34:25 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850865, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
Jun 19 06:34:25 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850865, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.253@KRBTEST.COM
Jun 19 06:34:25 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850865, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.252@KRBTEST.COM
I20260619 06:34:25.462791 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:25.426935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:41667 (local address 127.28.104.254:35203)
0619 06:34:25.434892 (+ 7957us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:25.434898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:25.436259 (+ 1361us) server_negotiation.cc:407] Connection header received
0619 06:34:25.440820 (+ 4561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:25.440829 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:25.440926 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:25.441052 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:25.444016 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.445171 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.448422 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.449132 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.449234 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:25.454598 (+ 5364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:25.454605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:25.454609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:25.454666 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:25.458799 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:25.459821 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:25.459826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:25.459830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:25.459992 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:25.460535 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:25.460541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:25.460545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:25.460943 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:25.461083 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:25.462177 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0619 06:34:25.462406 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7789,"thread_start_us":7653,"threads_started":1}
I20260619 06:34:25.462994 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:25.427783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.28.104.195:41285)
0619 06:34:25.437587 (+ 9804us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:25.437596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:25.437626 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:34:25.437699 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:25.437712 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:25.438024 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:25.438223 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:25.441291 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.443013 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.445104 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.446018 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.446166 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:25.454091 (+ 7925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:25.454103 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:25.454117 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:25.454152 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:25.458317 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:25.459307 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:25.459318 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:25.459322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:25.459502 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:25.459988 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:25.459996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:25.460002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:25.460499 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:25.460806 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:25.462186 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0619 06:34:25.462490 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9603,"thread_start_us":9403,"threads_started":1}
I20260619 06:34:25.464020 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:25.424125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:57427 (local address 127.28.104.252:33477)
0619 06:34:25.441153 (+ 17028us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:25.441160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:25.441178 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:34:25.441234 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:25.441239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:25.441310 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:25.441405 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:25.444046 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.445186 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.448454 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.449380 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.449525 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:25.455749 (+ 6224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:25.455757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:25.455764 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:25.455836 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:25.459712 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:25.460709 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:25.460714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:25.460718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:25.460945 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:25.461503 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:25.461508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:25.461511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:25.461886 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:25.462081 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:25.463362 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0619 06:34:25.463589 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16904,"thread_start_us":16779,"threads_started":1}
I20260619 06:34:25.464361 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:25.427267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.194:41667)
0619 06:34:25.435364 (+ 8097us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:25.435380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:25.440640 (+ 5260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:25.441522 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:25.441537 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:25.442571 (+ 1034us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:25.443808 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:25.443835 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.445352 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.445357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:25.448251 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:25.448265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.449542 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.449549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:25.449924 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:25.450695 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:25.450715 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:25.454352 (+ 3637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:25.458975 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:25.458985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:25.459005 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:25.459676 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:25.460145 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:25.460150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:25.460153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:25.460380 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:25.461179 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:25.461194 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:25.461672 (+ 478us) client_negotiation.cc:769] Sending connection context
0619 06:34:25.463051 (+ 1379us) client_negotiation.cc:240] Negotiation successful
0619 06:34:25.463310 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7975,"mutex_wait_us":636,"spinlock_wait_cycles":42496,"thread_start_us":7438,"threads_started":1}
I20260619 06:34:25.464512 3253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:25.466013 3928 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35203
I20260619 06:34:25.466396 3928 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:25.467191 3928 heartbeater.cc:511] Master 127.28.104.254:35203 requested a full tablet report, sending...
I20260619 06:34:25.468775 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:25.423950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:57655 (local address 127.28.104.253:46139)
0619 06:34:25.441426 (+ 17476us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:25.441435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:25.441462 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:34:25.441698 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:25.441704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:25.441802 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:25.441992 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:25.444025 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.445156 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.448421 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.449142 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.449246 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:25.455758 (+ 6512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:25.455764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:25.455769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:25.455822 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:25.459682 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:25.460595 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:25.460602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:25.460608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:25.460800 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:25.461269 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:25.461274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:25.461278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:25.461636 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:25.461887 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:25.468039 (+ 6152us) server_negotiation.cc:299] Negotiation successful
0619 06:34:25.468316 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17334,"thread_start_us":17560,"threads_started":1}
I20260619 06:34:25.468880 2797 ts_manager.cc:194] Re-registered known tserver with Master: c43f378aaf1940cc800b33e299cde2aa (127.28.104.194:45547)
I20260619 06:34:25.470197 2797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:41667
I20260619 06:34:25.471370 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:25.424567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:33477 (local address 127.28.104.194:57427)
0619 06:34:25.435207 (+ 10640us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:25.435265 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:25.440187 (+ 4922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:25.442268 (+ 2081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:25.442274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:25.442751 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:25.443808 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:25.443838 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.445367 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.445375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:25.448252 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:25.448270 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.449823 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.449831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:25.450190 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:25.450912 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:25.450945 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:25.455486 (+ 4541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:25.459918 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:25.459924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:25.459928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:25.460557 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:25.461129 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:25.461134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:25.461138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:25.461347 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:25.462214 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:25.462219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:25.462397 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 06:34:25.462545 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 06:34:25.471073 (+ 8528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10242,"mutex_wait_us":8572,"thread_start_us":13784,"threads_started":1}
I20260619 06:34:25.471786 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:25.424567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:46139 (local address 127.28.104.194:57655)
0619 06:34:25.437671 (+ 13104us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:25.437689 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:25.441502 (+ 3813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:25.442013 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:25.442017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:25.442606 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:25.443808 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:25.443835 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.445352 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.445357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:25.448251 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:25.448265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:25.449542 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:25.449549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:25.450304 (+ 755us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:25.451067 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:25.451088 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:25.455519 (+ 4431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:25.459820 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:25.459825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:25.459829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:25.460465 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:25.460944 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:25.460949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:25.460952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:25.461143 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:25.461971 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:25.461975 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:25.462131 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 06:34:25.462289 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 06:34:25.462849 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12963,"mutex_wait_us":3824,"spinlock_wait_cycles":6400,"thread_start_us":7945,"threads_started":1}
I20260619 06:34:25.472702 3927 heartbeater.cc:347] Connected to a master server at 127.28.104.252:33477
I20260619 06:34:25.472843 3927 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:25.473158 3929 heartbeater.cc:347] Connected to a master server at 127.28.104.253:46139
I20260619 06:34:25.473295 3929 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:25.473471 3927 heartbeater.cc:511] Master 127.28.104.252:33477 requested a full tablet report, sending...
I20260619 06:34:25.473728 3929 heartbeater.cc:511] Master 127.28.104.253:46139 requested a full tablet report, sending...
I20260619 06:34:25.474601 3596 ts_manager.cc:194] Registered new tserver with Master: c43f378aaf1940cc800b33e299cde2aa (127.28.104.194:45547)
I20260619 06:34:25.474877 3408 ts_manager.cc:194] Registered new tserver with Master: c43f378aaf1940cc800b33e299cde2aa (127.28.104.194:45547)
I20260619 06:34:25.554776 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 3129
I20260619 06:34:25.571205 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:41285
--local_ip_for_outbound_sockets=127.28.104.195
--tserver_master_addrs=127.28.104.254:35203
--webserver_port=40085
--webserver_interface=127.28.104.195
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab
--principal=oryx/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.104.254:35203,127.28.104.253:46139
--tserver_master_addrs=127.28.104.254:35203,127.28.104.253:46139,127.28.104.252:33477 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:34:25.825541 3778 heartbeater.cc:503] Master 127.28.104.254:35203 was elected leader, sending a full tablet report...
W20260619 06:34:25.914489 3940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:25.914816 3940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:25.914995 3940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:25.922560 3940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:25.922708 3940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:25.922874 3940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:34:25.933252 3940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab
--principal=oryx/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:41285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=40085
--tserver_master_addrs=127.28.104.254:35203,127.28.104.253:46139,127.28.104.252:33477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:25.934633 3940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:25.935971 3940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:25.949405 3946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:25.949615 3949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:25.949556 3947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:25.950819 3940 server_base.cc:1034] running on GCE node
I20260619 06:34:25.951551 3940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:25.952668 3940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:25.953915 3940 hybrid_clock.cc:648] HybridClock initialized: now 1781850865953869 us; error 43 us; skew 500 ppm
I20260619 06:34:25.954272 3940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:25 dist-test-slave-nf4n krb5kdc[2752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850865, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:25.958050 3940 init.cc:377] Logged in from keytab as oryx/127.28.104.195@KRBTEST.COM (short username oryx)
I20260619 06:34:25.959568 3940 webserver.cc:460] Webserver started at http://127.28.104.195:40085/ using document root <none> and password file <none>
I20260619 06:34:25.960338 3940 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:25.960490 3940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:25.966203 3940 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:34:25.968941 3956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:25.970382 3940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:34:25.970561 3940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "e688da4658c34574bc5a3f2a7f01417f"
format_stamp: "Formatted at 2026-06-19 06:34:22 on dist-test-slave-nf4n"
server_key: "dc18c473a8627a9725fbbc3ad2e8cba8"
server_key_iv: "a60b7be7fa81e645842972f39d2e3ddf"
server_key_version: "encryptionkey@0"
I20260619 06:34:25.971544 3940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:26.004349 3940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:26.008842 3940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:26.009219 3940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:26.010279 3940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:26.011722 3940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:26.011807 3940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:26.011924 3940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:26.011988 3940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:26.045622 3940 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:41285
I20260619 06:34:26.045712 4069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:41285 every 8 connection(s)
I20260619 06:34:26.047084 3940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:34:26.053402 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 3940
Jun 19 06:34:26 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850865, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.252@KRBTEST.COM
Jun 19 06:34:26 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850865, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.253@KRBTEST.COM
Jun 19 06:34:26 dist-test-slave-nf4n krb5kdc[2752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850865, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:26.102576 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.051295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:33477 (local address 127.28.104.195:58693)
0619 06:34:26.069538 (+ 18243us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:26.069564 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:26.069730 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:26.070728 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:26.070745 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:26.071902 (+ 1157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:26.072943 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.072974 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.075101 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.075109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.077616 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.077628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.078992 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.078997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.079232 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.083034 (+ 3802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:26.083074 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:26.092229 (+ 9155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:26.096172 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:26.096178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:26.096182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:26.097188 (+ 1006us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:26.097760 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:26.097768 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:26.097774 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:26.097973 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:26.098901 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:26.098914 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:26.099516 (+ 602us) client_negotiation.cc:769] Sending connection context
0619 06:34:26.099915 (+ 399us) client_negotiation.cc:240] Negotiation successful
0619 06:34:26.100676 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18067,"thread_start_us":16801,"threads_started":1}
I20260619 06:34:26.104059 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.051721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:58693 (local address 127.28.104.252:33477)
0619 06:34:26.052222 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.052232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.069680 (+ 17448us) server_negotiation.cc:407] Connection header received
0619 06:34:26.069887 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.069896 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.070010 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.070156 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:26.073254 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.074761 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.077832 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.078515 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.078626 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.092457 (+ 13831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:26.092464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:26.092469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:26.092509 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:26.096017 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.097384 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.097393 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.097399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.097604 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.098124 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.098132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.098137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.098625 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:26.098801 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.103196 (+ 4395us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.103472 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260619 06:34:26.105360 4070 heartbeater.cc:347] Connected to a master server at 127.28.104.252:33477
I20260619 06:34:26.105729 4070 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:26.106669 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.050894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.195:39553)
0619 06:34:26.070263 (+ 19369us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:26.070281 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:26.070362 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:26.077746 (+ 7384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:26.077752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:26.078417 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:26.078789 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.078804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.081430 (+ 2626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.081436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.082774 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.082791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.085562 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.085570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.085749 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.086367 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:26.086393 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:26.088427 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:26.093934 (+ 5507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:26.093945 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:26.093969 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:26.094618 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:26.096106 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:26.096114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:26.096121 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:26.096411 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:26.105501 (+ 9090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:26.105506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:26.105681 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 06:34:26.105890 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 06:34:26.106191 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19231,"mutex_wait_us":435,"spinlock_wait_cycles":768,"thread_start_us":17858,"threads_started":1}
I20260619 06:34:26.106746 4070 heartbeater.cc:511] Master 127.28.104.252:33477 requested a full tablet report, sending...
I20260619 06:34:26.107379 4074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.050108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:39553 (local address 127.28.104.254:35203)
0619 06:34:26.077325 (+ 27217us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.077335 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.077360 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:34:26.077449 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.077457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.077554 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.077697 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:26.079280 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.081303 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.084727 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.085378 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.085504 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.088955 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:26.088962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:26.088967 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:26.089005 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:26.093807 (+ 4802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.095812 (+ 2005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.095820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.095825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.095991 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.104843 (+ 8852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.104852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.104858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.105361 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:26.105523 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.106592 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.106858 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27046,"thread_start_us":27112,"threads_started":1}
I20260619 06:34:26.108728 3596 ts_manager.cc:194] Registered new tserver with Master: e688da4658c34574bc5a3f2a7f01417f (127.28.104.195:41285)
I20260619 06:34:26.108938 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.050894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:46139 (local address 127.28.104.195:54163)
0619 06:34:26.064149 (+ 13255us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:26.064220 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:26.067572 (+ 3352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:26.077952 (+ 10380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:26.077959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:26.078644 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:26.079047 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.079062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.080352 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.080358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.081563 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.081580 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.082945 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.082955 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.083104 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.083776 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:26.083802 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:26.088365 (+ 4563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:26.096179 (+ 7814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:26.096184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:26.096190 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:26.096872 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:26.105605 (+ 8733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:26.105609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:26.105614 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:26.105823 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:26.107645 (+ 1822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:26.107652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:26.107819 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 06:34:26.107962 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 06:34:26.108325 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12798,"spinlock_wait_cycles":58624,"thread_start_us":18966,"threads_started":1}
I20260619 06:34:26.109269 4071 heartbeater.cc:347] Connected to a master server at 127.28.104.254:35203
I20260619 06:34:26.109397 4071 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:26.109867 4071 heartbeater.cc:511] Master 127.28.104.254:35203 requested a full tablet report, sending...
I20260619 06:34:26.110324 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.050369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:54163 (local address 127.28.104.253:46139)
0619 06:34:26.076057 (+ 25688us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.076066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.076093 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:34:26.076170 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.076179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.076281 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.076405 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:26.079241 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.080193 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.081840 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.082508 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.082643 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.089176 (+ 6533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:26.089185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:26.089189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:26.089239 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:26.095610 (+ 6371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.104418 (+ 8808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.104428 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.104434 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.104674 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.105974 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.105979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.105983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.106369 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:26.109250 (+ 2881us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.109451 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.109782 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25521,"thread_start_us":25390,"threads_started":1}
I20260619 06:34:26.111373 2797 ts_manager.cc:194] Re-registered known tserver with Master: e688da4658c34574bc5a3f2a7f01417f (127.28.104.195:41285)
I20260619 06:34:26.112649 2797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:39553
I20260619 06:34:26.115516 4072 heartbeater.cc:347] Connected to a master server at 127.28.104.253:46139
I20260619 06:34:26.115644 4072 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:26.116032 4072 heartbeater.cc:511] Master 127.28.104.253:46139 requested a full tablet report, sending...
I20260619 06:34:26.117403 3408 ts_manager.cc:194] Registered new tserver with Master: e688da4658c34574bc5a3f2a7f01417f (127.28.104.195:41285)
I20260619 06:34:26.123226 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.087936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.28.104.252:33477)
0619 06:34:26.089114 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.089123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.093244 (+ 4121us) server_negotiation.cc:407] Connection header received
0619 06:34:26.093309 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.093315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.093404 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.093510 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:26.103626 (+ 10116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.105216 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.107756 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.108403 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.108498 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.112495 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:26.112502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:26.112507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:26.112543 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:26.119185 (+ 6642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.120289 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.120297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.120301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.120464 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.120932 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.120938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.120941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.121353 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:26.121613 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.122510 (+ 897us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.122725 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":814,"threads_started":1}
I20260619 06:34:26.127182 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.097056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.28.104.253:46139)
0619 06:34:26.110617 (+ 13561us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.110625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.110648 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:34:26.110710 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.110718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.110811 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.110905 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:26.113120 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.114447 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.116348 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.116982 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.117139 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.120772 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:26.120778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:26.120782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:26.120821 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:26.124264 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.125156 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.125159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.125162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.125281 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.125674 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.125678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.125681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.126049 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:26.126266 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.126625 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.126854 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13424,"thread_start_us":17724,"threads_started":1}
I20260619 06:34:26.127786 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.085732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.28.104.254:35203)
0619 06:34:26.089581 (+ 3849us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.089589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.089613 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:34:26.089678 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.089686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.089798 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.112487 (+ 22689us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:26.114609 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.115655 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.117725 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.118462 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.118583 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.121878 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:26.121882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:26.121886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:26.121917 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:26.124937 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.125692 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.125697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.125700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.125860 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.126230 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.126236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.126239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.126602 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:26.126854 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.127211 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.127457 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3704,"thread_start_us":10668,"threads_started":1}
I20260619 06:34:26.131548 2797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:34:26.138988 2797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:34:26.188051 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.160205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.28.104.193:37853)
0619 06:34:26.166994 (+ 6789us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.167008 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.176084 (+ 9076us) server_negotiation.cc:407] Connection header received
0619 06:34:26.179747 (+ 3663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.179762 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.180138 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.180341 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:26.181001 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.182631 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.185162 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.186797 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.187021 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.187234 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.187536 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.187730 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":569,"threads_started":1}
I20260619 06:34:26.188072 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.159639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:37853 (local address 127.0.0.1:45068)
0619 06:34:26.175884 (+ 16245us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:26.175903 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:26.179509 (+ 3606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:26.180316 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:26.180320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:26.180399 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:26.180816 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.180829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.182798 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.182803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.184994 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.185010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.187045 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.187057 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.187246 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.187289 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 06:34:26.187398 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 06:34:26.187588 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16117,"mutex_wait_us":2,"thread_start_us":15978,"threads_started":1}
I20260619 06:34:26.189764 4103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.159142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:45547 (local address 127.0.0.1:54530)
0619 06:34:26.172062 (+ 12920us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:26.172093 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:26.172296 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:26.182629 (+ 10333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:26.182635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:26.182665 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:26.183283 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.183299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.185142 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.185147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.187066 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.187078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.188934 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.188940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.189159 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.189193 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:34:26.189307 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 06:34:26.189436 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12741,"thread_start_us":18136,"threads_started":1}
I20260619 06:34:26.189801 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.159431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.28.104.194:45547)
0619 06:34:26.182056 (+ 22625us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.182065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.182092 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:34:26.182190 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.182197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.182421 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.182606 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:26.183490 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.184945 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.187238 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.188738 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.188918 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.189110 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.189367 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.189524 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22462,"thread_start_us":22117,"threads_started":1}
I20260619 06:34:26.192015 3710 tablet_service.cc:1467] Processing CreateTablet for tablet f79b4a1ddaf646249fe51d50115ab167 (DEFAULT_TABLE table=test-table [id=25015cea7c7645bc97af48b9cc28f7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:26.192966 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.161120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.28.104.195:41285)
0619 06:34:26.173685 (+ 12565us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.173699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.177671 (+ 3972us) server_negotiation.cc:407] Connection header received
0619 06:34:26.180897 (+ 3226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.180907 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.181185 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.181392 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:26.182062 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.183685 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.187342 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.188943 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.189169 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.189356 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.192476 (+ 3120us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.192648 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12310,"thread_start_us":12572,"threads_started":1}
I20260619 06:34:26.192998 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.160676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:41285 (local address 127.0.0.1:55468)
0619 06:34:26.177542 (+ 16866us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:26.177558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:26.179936 (+ 2378us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x61200002c628. stack: 00007fe42a70e66e 00007fe42a70e430 00007fe4297955ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fe42a7040fd 00007fe42a703d1e 00007fe43d60d3cd 00007fe43d60d36c 00007fe43d5d9b3b 00007fe431113536 00007fe431111d9e 00007fe431199dca 00007fe431197f68 00007fe4311cb82f
0619 06:34:26.179972 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:26.181379 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:26.181383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:26.181409 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:26.181744 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.181755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.183919 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.183924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.185766 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.185790 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.192054 (+ 6264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.192060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.192245 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.192301 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 06:34:26.192410 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 06:34:26.192550 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16740,"mutex_wait_us":4,"spinlock_wait_cycles":4228992,"thread_start_us":11015,"threads_started":1}
I20260619 06:34:26.192744 3861 tablet_service.cc:1467] Processing CreateTablet for tablet f79b4a1ddaf646249fe51d50115ab167 (DEFAULT_TABLE table=test-table [id=25015cea7c7645bc97af48b9cc28f7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:26.194190 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79b4a1ddaf646249fe51d50115ab167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:26.194293 3710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79b4a1ddaf646249fe51d50115ab167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:26.196568 4004 tablet_service.cc:1467] Processing CreateTablet for tablet f79b4a1ddaf646249fe51d50115ab167 (DEFAULT_TABLE table=test-table [id=25015cea7c7645bc97af48b9cc28f7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:26.197878 4004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79b4a1ddaf646249fe51d50115ab167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:26.207242 4107 tablet_bootstrap.cc:492] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: Bootstrap starting.
I20260619 06:34:26.210086 4108 tablet_bootstrap.cc:492] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: Bootstrap starting.
I20260619 06:34:26.212785 4109 tablet_bootstrap.cc:492] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: Bootstrap starting.
I20260619 06:34:26.212836 4107 tablet_bootstrap.cc:654] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:26.214322 4108 tablet_bootstrap.cc:654] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:26.214288 4107 log.cc:824] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:26.215533 4108 log.cc:824] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:26.216106 4109 tablet_bootstrap.cc:654] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:26.217337 4109 log.cc:824] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:26.217631 4107 tablet_bootstrap.cc:492] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: No bootstrap required, opened a new log
I20260619 06:34:26.217948 4107 ts_tablet_manager.cc:1397] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260619 06:34:26.218735 4108 tablet_bootstrap.cc:492] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: No bootstrap required, opened a new log
I20260619 06:34:26.219053 4108 ts_tablet_manager.cc:1397] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 06:34:26.220093 4109 tablet_bootstrap.cc:492] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: No bootstrap required, opened a new log
I20260619 06:34:26.220422 4109 ts_tablet_manager.cc:1397] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260619 06:34:26.227730 4109 raft_consensus.cc:348] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:26.228185 4109 raft_consensus.cc:374] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:26.228284 4109 raft_consensus.cc:729] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e688da4658c34574bc5a3f2a7f01417f, State: Initialized, Role: FOLLOWER
I20260619 06:34:26.228932 4109 consensus_queue.cc:260] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:26.230165 4071 heartbeater.cc:503] Master 127.28.104.254:35203 was elected leader, sending a full tablet report...
I20260619 06:34:26.230366 4108 raft_consensus.cc:348] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:26.230820 4108 raft_consensus.cc:374] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:26.230924 4108 raft_consensus.cc:729] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43f378aaf1940cc800b33e299cde2aa, State: Initialized, Role: FOLLOWER
I20260619 06:34:26.231645 4108 consensus_queue.cc:260] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:26.233331 4109 ts_tablet_manager.cc:1428] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: Time spent starting tablet: real 0.013s user 0.008s sys 0.002s
I20260619 06:34:26.239008 4107 raft_consensus.cc:348] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:26.239672 4107 raft_consensus.cc:374] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:26.239816 4107 raft_consensus.cc:729] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fad813d2f6144bda0cba9417d0af8b8, State: Initialized, Role: FOLLOWER
I20260619 06:34:26.240666 4107 consensus_queue.cc:260] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:26.243009 3928 heartbeater.cc:503] Master 127.28.104.254:35203 was elected leader, sending a full tablet report...
I20260619 06:34:26.245628 4108 ts_tablet_manager.cc:1428] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: Time spent starting tablet: real 0.026s user 0.013s sys 0.010s
I20260619 06:34:26.247884 4107 ts_tablet_manager.cc:1428] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: Time spent starting tablet: real 0.030s user 0.013s sys 0.002s
W20260619 06:34:26.274066 3780 tablet.cc:2369] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:34:26.318835 4073 tablet.cc:2369] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:34:26.329613 4114 raft_consensus.cc:482] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:34:26.329874 4114 raft_consensus.cc:504] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:26.331615 4114 leader_election.cc:288] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fad813d2f6144bda0cba9417d0af8b8 (127.28.104.193:37853), e688da4658c34574bc5a3f2a7f01417f (127.28.104.195:41285)
I20260619 06:34:26.338424 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.332114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:35687 (local address 127.28.104.195:41285)
0619 06:34:26.332370 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.332380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.333271 (+ 891us) server_negotiation.cc:407] Connection header received
0619 06:34:26.333499 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.333502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.333562 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.333659 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:26.334216 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.334920 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.336684 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.337513 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.337693 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.337769 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.338055 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.338177 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:34:26.338454 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.332163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:51397 (local address 127.28.104.193:37853)
0619 06:34:26.332499 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.332507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.333320 (+ 813us) server_negotiation.cc:407] Connection header received
0619 06:34:26.333531 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.333535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.333594 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.333690 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:26.334282 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.334996 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.336682 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.337517 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.337718 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.337797 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.338092 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.338215 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 06:34:26.338581 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.331898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:41285 (local address 127.28.104.194:35687)
0619 06:34:26.333154 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:26.333171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:26.333351 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:26.333673 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:26.333678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:26.333701 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:26.334072 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.334084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.335049 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.335056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.336515 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.336527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.337720 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.337726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.337886 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.337927 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:34:26.338029 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 06:34:26.338158 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":5,"thread_start_us":812,"threads_started":1}
I20260619 06:34:26.338934 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.331898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:37853 (local address 127.28.104.194:51397)
0619 06:34:26.333208 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:26.333224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:26.333373 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:26.333701 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:26.333706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:26.333737 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:26.334117 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.334125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.335131 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.335136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.336522 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:26.336534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.337777 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.337783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:26.337943 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.337965 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:34:26.338066 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:34:26.338188 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"mutex_wait_us":83,"thread_start_us":874,"threads_started":1}
I20260619 06:34:26.339637 4024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79b4a1ddaf646249fe51d50115ab167" candidate_uuid: "c43f378aaf1940cc800b33e299cde2aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e688da4658c34574bc5a3f2a7f01417f" is_pre_election: true
I20260619 06:34:26.340003 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79b4a1ddaf646249fe51d50115ab167" candidate_uuid: "c43f378aaf1940cc800b33e299cde2aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fad813d2f6144bda0cba9417d0af8b8" is_pre_election: true
I20260619 06:34:26.340132 4024 raft_consensus.cc:2454] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c43f378aaf1940cc800b33e299cde2aa in term 0.
I20260619 06:34:26.340458 3731 raft_consensus.cc:2454] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c43f378aaf1940cc800b33e299cde2aa in term 0.
I20260619 06:34:26.340827 3815 leader_election.cc:302] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c43f378aaf1940cc800b33e299cde2aa, e688da4658c34574bc5a3f2a7f01417f; no voters:
I20260619 06:34:26.341316 4114 raft_consensus.cc:2781] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:34:26.341459 4114 raft_consensus.cc:482] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:34:26.341586 4114 raft_consensus.cc:3037] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:26.343760 4114 raft_consensus.cc:504] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:26.344398 4114 leader_election.cc:288] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [CANDIDATE]: Term 1 election: Requested vote from peers 4fad813d2f6144bda0cba9417d0af8b8 (127.28.104.193:37853), e688da4658c34574bc5a3f2a7f01417f (127.28.104.195:41285)
I20260619 06:34:26.344817 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79b4a1ddaf646249fe51d50115ab167" candidate_uuid: "c43f378aaf1940cc800b33e299cde2aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fad813d2f6144bda0cba9417d0af8b8"
I20260619 06:34:26.344889 4024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79b4a1ddaf646249fe51d50115ab167" candidate_uuid: "c43f378aaf1940cc800b33e299cde2aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e688da4658c34574bc5a3f2a7f01417f"
I20260619 06:34:26.345007 3731 raft_consensus.cc:3037] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:26.345148 4024 raft_consensus.cc:3037] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:26.347196 3731 raft_consensus.cc:2454] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c43f378aaf1940cc800b33e299cde2aa in term 1.
I20260619 06:34:26.347340 4024 raft_consensus.cc:2454] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c43f378aaf1940cc800b33e299cde2aa in term 1.
I20260619 06:34:26.347631 3813 leader_election.cc:302] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fad813d2f6144bda0cba9417d0af8b8, c43f378aaf1940cc800b33e299cde2aa; no voters:
I20260619 06:34:26.347909 4114 raft_consensus.cc:2781] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:26.348340 4114 raft_consensus.cc:686] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 1 LEADER]: Becoming Leader. State: Replica: c43f378aaf1940cc800b33e299cde2aa, State: Running, Role: LEADER
I20260619 06:34:26.348907 4114 consensus_queue.cc:237] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:26.354403 2797 catalog_manager.cc:5495] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa reported cstate change: term changed from 0 to 1, leader changed from <none> to c43f378aaf1940cc800b33e299cde2aa (127.28.104.194). New cstate: current_term: 1 leader_uuid: "c43f378aaf1940cc800b33e299cde2aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } health_report { overall_health: UNKNOWN } } }
I20260619 06:34:26.375574 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.370118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.28.104.194:45547)
0619 06:34:26.370321 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.370327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.370348 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:34:26.370545 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.370549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.370627 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.370782 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:34:26.371170 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.371892 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.373235 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.373628 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.373765 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.374255 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:34:26.374851 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:34:26.375012 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.375063 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.375145 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:34:26.383962 3731 raft_consensus.cc:1261] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8 [term 1 FOLLOWER]: Refusing update from remote peer c43f378aaf1940cc800b33e299cde2aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:26.383972 4024 raft_consensus.cc:1261] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f [term 1 FOLLOWER]: Refusing update from remote peer c43f378aaf1940cc800b33e299cde2aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:26.384886 4118 consensus_queue.cc:1035] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:26.385275 4114 consensus_queue.cc:1035] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:26.397626 4124 mvcc.cc:205] Tried to move back new op lower bound from 7298461148699467776 to 7298461148569296896. Current Snapshot: MvccSnapshot[applied={T|T < 7298461148699467776}]
I20260619 06:34:26.421931 2796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:26.422292 2796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:26.434464 2796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 573e48de9df3491b85efe17e27d6a911: Sending DeleteTablet for 3 replicas of tablet f79b4a1ddaf646249fe51d50115ab167
I20260619 06:34:26.435642 3710 tablet_service.cc:1514] Processing DeleteTablet for tablet f79b4a1ddaf646249fe51d50115ab167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:26 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:45068
I20260619 06:34:26.435705 3861 tablet_service.cc:1514] Processing DeleteTablet for tablet f79b4a1ddaf646249fe51d50115ab167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:26 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:54530
I20260619 06:34:26.436750 4133 tablet_replica.cc:330] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: stopping tablet replica
I20260619 06:34:26.437368 4133 raft_consensus.cc:2229] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:34:26.438153 4133 raft_consensus.cc:2258] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:26.440815 4133 ts_tablet_manager.cc:1905] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:26.441941 4132 tablet_replica.cc:330] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: stopping tablet replica
I20260619 06:34:26.442361 4132 raft_consensus.cc:2229] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:26.442847 4132 raft_consensus.cc:2258] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:26.445523 4004 tablet_service.cc:1514] Processing DeleteTablet for tablet f79b4a1ddaf646249fe51d50115ab167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:26 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:55468
I20260619 06:34:26.451359 4132 ts_tablet_manager.cc:1905] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:26.455564 4133 ts_tablet_manager.cc:1918] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:26.455758 4133 log.cc:1196] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/f79b4a1ddaf646249fe51d50115ab167
I20260619 06:34:26.456393 4133 ts_tablet_manager.cc:1939] T f79b4a1ddaf646249fe51d50115ab167 P c43f378aaf1940cc800b33e299cde2aa: Deleting consensus metadata
I20260619 06:34:26.458280 2784 catalog_manager.cc:4841] TS c43f378aaf1940cc800b33e299cde2aa (127.28.104.194:45547): tablet f79b4a1ddaf646249fe51d50115ab167 (table test-table [id=25015cea7c7645bc97af48b9cc28f7cc]) successfully deleted
I20260619 06:34:26.458781 4132 ts_tablet_manager.cc:1918] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:26.458958 4132 log.cc:1196] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/f79b4a1ddaf646249fe51d50115ab167
I20260619 06:34:26.459630 4135 tablet_replica.cc:330] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: stopping tablet replica
I20260619 06:34:26.459897 4132 ts_tablet_manager.cc:1939] T f79b4a1ddaf646249fe51d50115ab167 P 4fad813d2f6144bda0cba9417d0af8b8: Deleting consensus metadata
I20260619 06:34:26.460134 4135 raft_consensus.cc:2229] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:26.460243 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 2765
I20260619 06:34:26.460645 4135 raft_consensus.cc:2258] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:26.461501 2827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:26.465233 4135 ts_tablet_manager.cc:1905] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:26.472512 4135 ts_tablet_manager.cc:1918] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:26.472703 4135 log.cc:1196] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/f79b4a1ddaf646249fe51d50115ab167
I20260619 06:34:26.473351 4135 ts_tablet_manager.cc:1939] T f79b4a1ddaf646249fe51d50115ab167 P e688da4658c34574bc5a3f2a7f01417f: Deleting consensus metadata
I20260619 06:34:26.478124 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.438393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.28.104.253:46139)
0619 06:34:26.438579 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.438586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.458854 (+ 20268us) server_negotiation.cc:407] Connection header received
0619 06:34:26.461132 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.461138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.461248 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.461330 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:26.463259 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.464363 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.466256 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.466901 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.467002 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.470630 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:26.470639 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:26.470644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:26.470687 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:26.474300 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.475186 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.475192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.475197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.475378 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.475910 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.475916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.475920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.476306 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:26.476571 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.477361 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.477738 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:34:26.494226 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.459276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.28.104.252:33477)
0619 06:34:26.459464 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.459470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.478138 (+ 18668us) server_negotiation.cc:407] Connection header received
0619 06:34:26.478381 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.478388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.478484 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.478553 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:26.480412 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.481641 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.483516 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.484209 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.484343 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.487647 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:26.487654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:26.487659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:26.487702 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:26.491245 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.492119 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.492124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.492128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.492276 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.492707 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.492712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.492715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.493123 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:26.493379 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.493654 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.493877 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:34:26.602910 2781 catalog_manager.cc:4841] TS 4fad813d2f6144bda0cba9417d0af8b8 (127.28.104.193:37853): tablet f79b4a1ddaf646249fe51d50115ab167 (table test-table [id=25015cea7c7645bc97af48b9cc28f7cc]) successfully deleted
I20260619 06:34:26.602937 2783 catalog_manager.cc:4841] TS e688da4658c34574bc5a3f2a7f01417f (127.28.104.195:41285): tablet f79b4a1ddaf646249fe51d50115ab167 (table test-table [id=25015cea7c7645bc97af48b9cc28f7cc]) successfully deleted
W20260619 06:34:26.603163 2781 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.104.193:37853, user_credentials={real_user=slave}} blocked reactor thread for 141364us
I20260619 06:34:26.604156 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 2765
I20260619 06:34:26.666389 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.639552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.28.104.252:33477)
0619 06:34:26.639756 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.639762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.646760 (+ 6998us) server_negotiation.cc:407] Connection header received
0619 06:34:26.647344 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.647355 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.647470 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.647561 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:26.650620 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.651727 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.654068 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.654723 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.654819 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.659494 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:26.659503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:26.659510 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:26.659557 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:26.663260 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.664146 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.664152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.664157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.664317 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.664727 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.664732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.664736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.665108 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:26.665354 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.665743 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.665986 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:34:26.668380 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:26.639943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.28.104.253:46139)
0619 06:34:26.640165 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:26.640174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:26.650819 (+ 10645us) server_negotiation.cc:407] Connection header received
0619 06:34:26.651262 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:26.651269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:26.651347 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:26.651413 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:26.653935 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.655001 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.656985 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:26.657439 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:26.657578 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:26.660261 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:26.660270 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:26.660275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:26.660328 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:26.664497 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.665423 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.665428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.665431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.665584 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:26.666926 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:26.666931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:26.666935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:26.667296 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:26.667544 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:26.667911 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 06:34:26.668087 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:34:27.403694 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:27.402814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.193:49807)
0619 06:34:27.403330 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:27.403528 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":250,"threads_started":1}
W20260619 06:34:27.404116 3778 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:35203 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
I20260619 06:34:27.412847 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:27.412160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.194:37373)
0619 06:34:27.412593 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:27.412693 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":258,"threads_started":1}
W20260619 06:34:27.413267 3928 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:35203 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
I20260619 06:34:27.415722 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:27.414837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.195:42381)
0619 06:34:27.415315 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:27.415485 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":255,"threads_started":1}
W20260619 06:34:27.416101 4071 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:35203 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
I20260619 06:34:27.933714 4147 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 573e48de9df3491b85efe17e27d6a911)
I20260619 06:34:27.934017 4147 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } }
I20260619 06:34:27.935678 4147 leader_election.cc:288] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 573e48de9df3491b85efe17e27d6a911 (127.28.104.254:35203), d9102c0e7dfe4c00bae643db3ddc784f (127.28.104.253:46139)
I20260619 06:34:27.936900 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:27.935803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:40464)
0619 06:34:27.936391 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:27.936610 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":306,"threads_started":1}
W20260619 06:34:27.937229 3583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
I20260619 06:34:27.938772 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:27.938387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:40466)
0619 06:34:27.938547 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:27.938629 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 06:34:27.938961 3583 leader_election.cc:334] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 573e48de9df3491b85efe17e27d6a911 (127.28.104.254:35203): Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
I20260619 06:34:27.942453 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:27.936016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:46139 (local address 127.0.0.1:59006)
0619 06:34:27.936943 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:27.936957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:27.937149 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:27.937532 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:27.937537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:27.937564 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:27.937893 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:27.937902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:27.939177 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:27.939184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:27.940479 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:27.940490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:27.941647 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:27.941653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:27.941821 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:27.941845 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:34:27.941953 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 06:34:27.942083 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":681,"threads_started":1}
I20260619 06:34:27.942448 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:27.936104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.28.104.253:46139)
0619 06:34:27.936665 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:27.936673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:27.937079 (+ 406us) server_negotiation.cc:407] Connection header received
0619 06:34:27.937297 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:27.937301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:27.937360 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:27.937490 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:27.938080 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:27.939044 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:27.940633 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:27.941477 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:27.941652 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:27.941728 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:27.941979 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 06:34:27.942114 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":258,"threads_started":1}
I20260619 06:34:27.943169 3418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa378990b19c4102b7d801f2c71a7a41" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" is_pre_election: true
I20260619 06:34:27.943548 3418 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa378990b19c4102b7d801f2c71a7a41 in term 1.
I20260619 06:34:27.944060 3582 leader_election.cc:302] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aa378990b19c4102b7d801f2c71a7a41, d9102c0e7dfe4c00bae643db3ddc784f; no voters: 573e48de9df3491b85efe17e27d6a911
I20260619 06:34:27.944481 4147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 06:34:27.944612 4147 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 573e48de9df3491b85efe17e27d6a911)
I20260619 06:34:27.944715 4147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:34:27.946973 4147 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } }
I20260619 06:34:27.947752 4147 leader_election.cc:288] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [CANDIDATE]: Term 2 election: Requested vote from peers 573e48de9df3491b85efe17e27d6a911 (127.28.104.254:35203), d9102c0e7dfe4c00bae643db3ddc784f (127.28.104.253:46139)
I20260619 06:34:27.948349 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:27.947829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:40482)
0619 06:34:27.948093 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:27.948196 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260619 06:34:27.948527 3418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa378990b19c4102b7d801f2c71a7a41" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d9102c0e7dfe4c00bae643db3ddc784f"
I20260619 06:34:27.948748 3418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:34:27.949556 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:27.949140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:40490)
0619 06:34:27.949300 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:27.949399 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 06:34:27.949779 3583 leader_election.cc:334] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 573e48de9df3491b85efe17e27d6a911 (127.28.104.254:35203): Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
I20260619 06:34:27.951166 3418 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa378990b19c4102b7d801f2c71a7a41 in term 2.
I20260619 06:34:27.951654 3582 leader_election.cc:302] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aa378990b19c4102b7d801f2c71a7a41, d9102c0e7dfe4c00bae643db3ddc784f; no voters: 573e48de9df3491b85efe17e27d6a911
I20260619 06:34:27.951982 4147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:34:27.952468 4147 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [term 2 LEADER]: Becoming Leader. State: Replica: aa378990b19c4102b7d801f2c71a7a41, State: Running, Role: LEADER
I20260619 06:34:27.952853 4147 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } }
I20260619 06:34:27.955610 4152 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa378990b19c4102b7d801f2c71a7a41. Latest consensus state: current_term: 2 leader_uuid: "aa378990b19c4102b7d801f2c71a7a41" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:27.955951 4152 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:27.956552 4154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:27.960193 4154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:27.961015 4154 catalog_manager.cc:1259] Loaded cluster ID: 7951725ab48a4ccfae5b607186e2a0c7
I20260619 06:34:27.961165 4154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:27.962437 4154 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:27.963179 4154 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Loaded TSK: 0
I20260619 06:34:27.964011 4154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:34:28.004467 3595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:34:28.011883 3595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:34:28.016500 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.016053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:40504)
0619 06:34:28.016227 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.016325 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:34:28.016526 3418 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [term 2 FOLLOWER]: Refusing update from remote peer aa378990b19c4102b7d801f2c71a7a41: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 06:34:28.017462 4152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 06:34:28.017624 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.017194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:40506)
0619 06:34:28.017378 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.017446 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 06:34:28.017805 3583 consensus_peers.cc:487] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 -> Peer 573e48de9df3491b85efe17e27d6a911 (127.28.104.254:35203): Couldn't send request to peer 573e48de9df3491b85efe17e27d6a911. Status: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:34:28.020200 4156 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa378990b19c4102b7d801f2c71a7a41. Latest consensus state: current_term: 2 leader_uuid: "aa378990b19c4102b7d801f2c71a7a41" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:28.020649 4156 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:34:28.022153 4147 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "aa378990b19c4102b7d801f2c71a7a41" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:28.022497 4147 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:28.022817 4156 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "aa378990b19c4102b7d801f2c71a7a41" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "573e48de9df3491b85efe17e27d6a911" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 35203 } } peers { permanent_uuid: "d9102c0e7dfe4c00bae643db3ddc784f" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "aa378990b19c4102b7d801f2c71a7a41" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 33477 } attrs { promote: false } } }
I20260619 06:34:28.023070 4156 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9102c0e7dfe4c00bae643db3ddc784f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:34:28.023319 4157 mvcc.cc:205] Tried to move back new op lower bound from 7298461155388641280 to 7298461155138838528. Current Snapshot: MvccSnapshot[applied={T|T < 7298461155388641280 or (T in {7298461155388641280})}]
I20260619 06:34:28.037885 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.031558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:45547 (local address 127.0.0.1:54538)
0619 06:34:28.031677 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.031692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:28.031802 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:28.032543 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:28.032548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:28.032570 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:28.032849 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:28.032860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.034167 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.034173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:28.035578 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:28.035591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.037127 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.037135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:28.037344 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.037379 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:34:28.037453 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:34:28.037588 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 06:34:28.038655 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.031775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.28.104.194:45547)
0619 06:34:28.032268 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:28.032274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:28.032299 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:34:28.032355 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:28.032359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:28.032419 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:28.032532 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:28.033171 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.034027 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.035724 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.036722 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.038200 (+ 1478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.038267 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:28.038314 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 06:34:28.038449 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":256,"threads_started":1}
I20260619 06:34:28.039337 3861 tablet_service.cc:1467] Processing CreateTablet for tablet b961c5907cce45f6a98ceb9b327836c9 (DEFAULT_TABLE table=test-table [id=d322725f8bb34494b0f51c75aa0f439d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:28.039769 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b961c5907cce45f6a98ceb9b327836c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:28.044606 4165 tablet_bootstrap.cc:492] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa: Bootstrap starting.
I20260619 06:34:28.046352 4165 tablet_bootstrap.cc:654] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:28.048480 4165 tablet_bootstrap.cc:492] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa: No bootstrap required, opened a new log
I20260619 06:34:28.048635 4165 ts_tablet_manager.cc:1397] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 06:34:28.049454 4165 raft_consensus.cc:348] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:28.049690 4165 raft_consensus.cc:374] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:28.049774 4165 raft_consensus.cc:729] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43f378aaf1940cc800b33e299cde2aa, State: Initialized, Role: FOLLOWER
I20260619 06:34:28.050019 4165 consensus_queue.cc:260] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:28.050830 4165 ts_tablet_manager.cc:1428] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 06:34:28.053006 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.031168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:37853 (local address 127.0.0.1:45072)
0619 06:34:28.031341 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.031362 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:28.031473 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:28.046951 (+ 15478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:28.046956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:28.046988 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:28.047168 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:28.047183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.048732 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.048737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:28.050599 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:28.050615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.052215 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.052221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:28.052413 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.052464 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 06:34:28.052550 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:34:28.052688 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 06:34:28.054628 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.031324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.28.104.193:37853)
0619 06:34:28.046614 (+ 15290us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:28.046621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:28.046642 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:34:28.046711 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:28.046716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:28.046787 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:28.046899 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:28.047385 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.048560 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.050817 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.052036 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.053408 (+ 1372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.053522 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:28.053574 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:34:28.053698 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15160,"thread_start_us":15657,"threads_started":1}
I20260619 06:34:28.054744 3927 heartbeater.cc:503] Master 127.28.104.252:33477 was elected leader, sending a full tablet report...
I20260619 06:34:28.056049 3710 tablet_service.cc:1467] Processing CreateTablet for tablet b961c5907cce45f6a98ceb9b327836c9 (DEFAULT_TABLE table=test-table [id=d322725f8bb34494b0f51c75aa0f439d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:28.056560 3710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b961c5907cce45f6a98ceb9b327836c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:28.061918 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.051060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.194:60567)
0619 06:34:28.061541 (+ 10481us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.061675 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10381,"thread_start_us":10194,"threads_started":1}
I20260619 06:34:28.062423 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.032478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:41285 (local address 127.0.0.1:55470)
0619 06:34:28.033375 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.033394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:28.033502 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:28.054025 (+ 20523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:28.054030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:28.054057 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:28.054476 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:28.054493 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.055993 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.056001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:28.059838 (+ 3837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:28.059855 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.061640 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.061647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:28.061831 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.061869 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:34:28.061946 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:34:28.062085 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":654,"threads_started":1}
I20260619 06:34:28.062944 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.032951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.28.104.195:41285)
0619 06:34:28.053714 (+ 20763us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:28.053720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:28.053737 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:34:28.053802 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:28.053807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:28.053886 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:28.054010 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:28.054813 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.055803 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.060042 (+ 4239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.061409 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.062201 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.062275 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:28.062317 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 06:34:28.062637 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20645,"thread_start_us":21180,"threads_started":1}
I20260619 06:34:28.063026 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.062638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.194:34035)
0619 06:34:28.062832 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.062907 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 06:34:28.063675 4004 tablet_service.cc:1467] Processing CreateTablet for tablet b961c5907cce45f6a98ceb9b327836c9 (DEFAULT_TABLE table=test-table [id=d322725f8bb34494b0f51c75aa0f439d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:28.064079 4004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b961c5907cce45f6a98ceb9b327836c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:28.063815 4169 tablet_bootstrap.cc:492] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8: Bootstrap starting.
I20260619 06:34:28.065714 4169 tablet_bootstrap.cc:654] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:28.067601 4169 tablet_bootstrap.cc:492] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8: No bootstrap required, opened a new log
I20260619 06:34:28.067764 4169 ts_tablet_manager.cc:1397] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 06:34:28.068404 4169 raft_consensus.cc:348] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:28.068696 4169 raft_consensus.cc:374] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:28.068799 4169 raft_consensus.cc:729] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fad813d2f6144bda0cba9417d0af8b8, State: Initialized, Role: FOLLOWER
I20260619 06:34:28.068969 4171 tablet_bootstrap.cc:492] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f: Bootstrap starting.
I20260619 06:34:28.069097 4169 consensus_queue.cc:260] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:28.069943 4169 ts_tablet_manager.cc:1428] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 06:34:28.070843 4171 tablet_bootstrap.cc:654] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:28.073491 3777 heartbeater.cc:503] Master 127.28.104.252:33477 was elected leader, sending a full tablet report...
I20260619 06:34:28.073820 4171 tablet_bootstrap.cc:492] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f: No bootstrap required, opened a new log
I20260619 06:34:28.073968 4171 ts_tablet_manager.cc:1397] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:34:28.074368 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.070158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.193:57647)
0619 06:34:28.073927 (+ 3769us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.074116 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3613,"threads_started":1}
I20260619 06:34:28.074765 4171 raft_consensus.cc:348] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:28.074998 4171 raft_consensus.cc:374] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:28.075093 4171 raft_consensus.cc:729] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e688da4658c34574bc5a3f2a7f01417f, State: Initialized, Role: FOLLOWER
I20260619 06:34:28.075366 4171 consensus_queue.cc:260] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:28.075965 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.075301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.193:50191)
0619 06:34:28.075727 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.075835 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300}
I20260619 06:34:28.076220 4171 ts_tablet_manager.cc:1428] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:34:28.077451 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.076526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.195:44287)
0619 06:34:28.077087 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.077212 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":289,"threads_started":1}
I20260619 06:34:28.078037 4070 heartbeater.cc:503] Master 127.28.104.252:33477 was elected leader, sending a full tablet report...
I20260619 06:34:28.078981 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.078045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.195:43205)
0619 06:34:28.078602 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.078731 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":249,"threads_started":1}
I20260619 06:34:28.088706 4175 raft_consensus.cc:482] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:34:28.088922 4175 raft_consensus.cc:504] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:28.090556 4175 leader_election.cc:288] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fad813d2f6144bda0cba9417d0af8b8 (127.28.104.193:37853), c43f378aaf1940cc800b33e299cde2aa (127.28.104.194:45547)
I20260619 06:34:28.096174 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.090688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:37853 (local address 127.28.104.195:43987)
0619 06:34:28.090859 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.090878 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:28.091001 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:28.091453 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:28.091458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:28.091488 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:28.091797 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:28.091809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.092856 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.092860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:28.094302 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:28.094313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.095380 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.095386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:28.095593 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.095629 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:34:28.095756 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 06:34:28.095899 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":26}
I20260619 06:34:28.096196 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.090822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:43987 (local address 127.28.104.193:37853)
0619 06:34:28.091155 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:28.091162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:28.091183 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:34:28.091257 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:28.091263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:28.091338 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:28.091401 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:28.091941 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.092716 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.094449 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.095205 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.095357 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.095434 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:28.095798 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 06:34:28.095929 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:34:28.096535 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.091069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:41337 (local address 127.28.104.194:45547)
0619 06:34:28.091265 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:28.091269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:28.091283 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:34:28.091338 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:28.091342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:28.091437 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:28.091532 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:28.091991 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.092636 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.094495 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.095243 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.095424 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.095495 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:28.096119 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 06:34:28.096259 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:34:28.097141 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.090835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:45547 (local address 127.28.104.195:41337)
0619 06:34:28.091032 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.091056 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:28.091166 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:28.091563 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:28.091567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:28.091589 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:28.091825 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:28.091832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.092765 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.092772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:28.094336 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:28.094348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.095711 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.095736 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:28.095939 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.095969 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:34:28.096699 (+ 730us) client_negotiation.cc:240] Negotiation successful
0619 06:34:28.096855 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":98}
I20260619 06:34:28.097152 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b961c5907cce45f6a98ceb9b327836c9" candidate_uuid: "e688da4658c34574bc5a3f2a7f01417f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fad813d2f6144bda0cba9417d0af8b8" is_pre_election: true
I20260619 06:34:28.097436 3731 raft_consensus.cc:2454] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e688da4658c34574bc5a3f2a7f01417f in term 0.
I20260619 06:34:28.097913 3957 leader_election.cc:302] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fad813d2f6144bda0cba9417d0af8b8, e688da4658c34574bc5a3f2a7f01417f; no voters:
I20260619 06:34:28.097954 3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b961c5907cce45f6a98ceb9b327836c9" candidate_uuid: "e688da4658c34574bc5a3f2a7f01417f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c43f378aaf1940cc800b33e299cde2aa" is_pre_election: true
I20260619 06:34:28.098287 4175 raft_consensus.cc:2781] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:34:28.098371 3881 raft_consensus.cc:2454] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e688da4658c34574bc5a3f2a7f01417f in term 0.
I20260619 06:34:28.098465 4175 raft_consensus.cc:482] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:34:28.098565 4175 raft_consensus.cc:3037] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:28.100665 4175 raft_consensus.cc:504] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:28.101394 4175 leader_election.cc:288] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [CANDIDATE]: Term 1 election: Requested vote from peers 4fad813d2f6144bda0cba9417d0af8b8 (127.28.104.193:37853), c43f378aaf1940cc800b33e299cde2aa (127.28.104.194:45547)
I20260619 06:34:28.101794 3731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b961c5907cce45f6a98ceb9b327836c9" candidate_uuid: "e688da4658c34574bc5a3f2a7f01417f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fad813d2f6144bda0cba9417d0af8b8"
I20260619 06:34:28.101917 3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b961c5907cce45f6a98ceb9b327836c9" candidate_uuid: "e688da4658c34574bc5a3f2a7f01417f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c43f378aaf1940cc800b33e299cde2aa"
I20260619 06:34:28.102010 3731 raft_consensus.cc:3037] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:28.102108 3881 raft_consensus.cc:3037] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:28.105075 3731 raft_consensus.cc:2454] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e688da4658c34574bc5a3f2a7f01417f in term 1.
I20260619 06:34:28.105075 3881 raft_consensus.cc:2454] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e688da4658c34574bc5a3f2a7f01417f in term 1.
I20260619 06:34:28.105455 3957 leader_election.cc:302] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4fad813d2f6144bda0cba9417d0af8b8, c43f378aaf1940cc800b33e299cde2aa, e688da4658c34574bc5a3f2a7f01417f; no voters:
I20260619 06:34:28.105746 4175 raft_consensus.cc:2781] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:28.106185 4175 raft_consensus.cc:686] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 1 LEADER]: Becoming Leader. State: Replica: e688da4658c34574bc5a3f2a7f01417f, State: Running, Role: LEADER
I20260619 06:34:28.106616 4175 consensus_queue.cc:237] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } } peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } }
I20260619 06:34:28.107227 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.106770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.195:42745)
0619 06:34:28.106962 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.107052 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:34:28.110247 3596 catalog_manager.cc:5495] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f reported cstate change: term changed from 0 to 1, leader changed from <none> to e688da4658c34574bc5a3f2a7f01417f (127.28.104.195). New cstate: current_term: 1 leader_uuid: "e688da4658c34574bc5a3f2a7f01417f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e688da4658c34574bc5a3f2a7f01417f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 41285 } health_report { overall_health: HEALTHY } } }
I20260619 06:34:28.144228 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.139165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.28.104.195:41285)
0619 06:34:28.139381 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:28.139387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:28.139406 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:34:28.139568 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:28.139572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:28.139658 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:28.139807 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:34:28.140158 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.140857 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.142155 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.142567 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.142695 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.143114 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:34:28.143662 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:34:28.143821 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:28.143875 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 06:34:28.143955 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:34:28.148279 3731 raft_consensus.cc:1261] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8 [term 1 FOLLOWER]: Refusing update from remote peer e688da4658c34574bc5a3f2a7f01417f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:28.148590 3881 raft_consensus.cc:1261] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa [term 1 FOLLOWER]: Refusing update from remote peer e688da4658c34574bc5a3f2a7f01417f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:28.149202 4178 consensus_queue.cc:1035] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fad813d2f6144bda0cba9417d0af8b8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:28.149639 4175 consensus_queue.cc:1035] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c43f378aaf1940cc800b33e299cde2aa" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 45547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:28.152151 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.151679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.194:56589)
0619 06:34:28.151896 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.151999 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260619 06:34:28.152144 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.151628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.193:53987)
0619 06:34:28.151814 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.151922 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 06:34:28.157522 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.156895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.193:35943)
0619 06:34:28.157197 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.157310 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260619 06:34:28.161746 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.161202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.195:48277)
0619 06:34:28.161466 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.161586 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260619 06:34:28.170547 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.170004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.194:34101)
0619 06:34:28.170254 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.170351 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 06:34:28.170583 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.170076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.28.104.195:48881)
0619 06:34:28.170307 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.170407 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 06:34:28.170358 4184 mvcc.cc:205] Tried to move back new op lower bound from 7298461155928133632 to 7298461155769040896. Current Snapshot: MvccSnapshot[applied={T|T < 7298461155928133632}]
I20260619 06:34:28.175182 3595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:28.175459 3595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:28.181339 3595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa378990b19c4102b7d801f2c71a7a41: Sending DeleteTablet for 3 replicas of tablet b961c5907cce45f6a98ceb9b327836c9
I20260619 06:34:28.182637 3710 tablet_service.cc:1514] Processing DeleteTablet for tablet b961c5907cce45f6a98ceb9b327836c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:28 UTC) from {username='oryx', principal='oryx/127.28.104.252@KRBTEST.COM'} at 127.0.0.1:45072
I20260619 06:34:28.182885 3861 tablet_service.cc:1514] Processing DeleteTablet for tablet b961c5907cce45f6a98ceb9b327836c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:28 UTC) from {username='oryx', principal='oryx/127.28.104.252@KRBTEST.COM'} at 127.0.0.1:54538
I20260619 06:34:28.182878 4004 tablet_service.cc:1514] Processing DeleteTablet for tablet b961c5907cce45f6a98ceb9b327836c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:28 UTC) from {username='oryx', principal='oryx/127.28.104.252@KRBTEST.COM'} at 127.0.0.1:55470
I20260619 06:34:28.189654 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 3634
I20260619 06:34:28.195626 4193 tablet_replica.cc:330] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa: stopping tablet replica
I20260619 06:34:28.196025 4193 raft_consensus.cc:2229] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:28.196503 4193 raft_consensus.cc:2258] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:28.197220 4194 tablet_replica.cc:330] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f: stopping tablet replica
I20260619 06:34:28.197552 4194 raft_consensus.cc:2229] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:34:28.198134 4193 ts_tablet_manager.cc:1905] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:28.198297 4194 raft_consensus.cc:2258] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:28.198730 4192 tablet_replica.cc:330] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8: stopping tablet replica
I20260619 06:34:28.199290 4192 raft_consensus.cc:2229] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:28.199522 4194 ts_tablet_manager.cc:1905] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:28.199584 4192 raft_consensus.cc:2258] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:28.200641 4192 ts_tablet_manager.cc:1905] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:28.204944 4193 ts_tablet_manager.cc:1918] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:28.205113 4193 log.cc:1196] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/b961c5907cce45f6a98ceb9b327836c9
I20260619 06:34:28.205610 4193 ts_tablet_manager.cc:1939] T b961c5907cce45f6a98ceb9b327836c9 P c43f378aaf1940cc800b33e299cde2aa: Deleting consensus metadata
I20260619 06:34:28.206743 3583 catalog_manager.cc:4841] TS c43f378aaf1940cc800b33e299cde2aa (127.28.104.194:45547): tablet b961c5907cce45f6a98ceb9b327836c9 (table test-table [id=d322725f8bb34494b0f51c75aa0f439d]) successfully deleted
I20260619 06:34:28.207314 4192 ts_tablet_manager.cc:1918] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:28.207372 4194 ts_tablet_manager.cc:1918] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:28.207456 4192 log.cc:1196] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/b961c5907cce45f6a98ceb9b327836c9
I20260619 06:34:28.207546 4194 log.cc:1196] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/b961c5907cce45f6a98ceb9b327836c9
I20260619 06:34:28.207943 4194 ts_tablet_manager.cc:1939] T b961c5907cce45f6a98ceb9b327836c9 P e688da4658c34574bc5a3f2a7f01417f: Deleting consensus metadata
I20260619 06:34:28.207959 4192 ts_tablet_manager.cc:1939] T b961c5907cce45f6a98ceb9b327836c9 P 4fad813d2f6144bda0cba9417d0af8b8: Deleting consensus metadata
I20260619 06:34:28.208910 3582 catalog_manager.cc:4841] TS e688da4658c34574bc5a3f2a7f01417f (127.28.104.195:41285): tablet b961c5907cce45f6a98ceb9b327836c9 (table test-table [id=d322725f8bb34494b0f51c75aa0f439d]) successfully deleted
I20260619 06:34:28.209067 3580 catalog_manager.cc:4841] TS 4fad813d2f6144bda0cba9417d0af8b8 (127.28.104.193:37853): tablet b961c5907cce45f6a98ceb9b327836c9 (table test-table [id=d322725f8bb34494b0f51c75aa0f439d]) successfully deleted
I20260619 06:34:28.220532 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.190531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.28.104.193:37853)
0619 06:34:28.190733 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:28.190741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:28.200481 (+ 9740us) server_negotiation.cc:407] Connection header received
0619 06:34:28.200807 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:28.200813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:28.200892 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:28.200980 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:28.209164 (+ 8184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.210082 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.211320 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.211791 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.211922 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.214677 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:28.214695 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:28.214722 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:28.214769 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:28.217721 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:28.218400 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:28.218408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:28.218410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:28.218537 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:28.219071 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:28.219075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:28.219077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:28.219417 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:28.219640 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:28.220048 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 06:34:28.220229 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:34:28.221284 3772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:28.316026 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 3634
I20260619 06:34:28.340548 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 3789
I20260619 06:34:28.356081 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.341386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.28.104.194:45547)
0619 06:34:28.341582 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:28.341590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:28.341617 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:34:28.341737 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:28.341742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:28.341826 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:28.341945 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:28.343199 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.344309 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.345637 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.346275 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.346415 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.349136 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:28.349157 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:28.349178 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:28.349227 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:28.353374 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:28.353976 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:28.353986 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:28.353990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:28.354146 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:28.354496 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:28.354501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:28.354504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:28.354969 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:28.355272 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:28.355475 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 06:34:28.355695 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:34:28.356778 3922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:28.482165 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 3789
I20260619 06:34:28.504230 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 3940
I20260619 06:34:28.520744 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.505062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.28.104.195:41285)
0619 06:34:28.505257 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:28.505261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:28.505356 (+ 95us) server_negotiation.cc:407] Connection header received
0619 06:34:28.505544 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:28.505548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:28.505627 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:28.505713 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:28.507622 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.508462 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.510323 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:28.510880 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:28.510978 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:28.514453 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:28.514465 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:28.514480 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:28.514516 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:28.517721 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:28.518577 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:28.518603 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:28.518609 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:28.518770 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:28.519185 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:28.519188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:28.519190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:28.519640 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:28.519894 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:28.520232 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 06:34:28.520392 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:34:28.521579 4065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:28.530002 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.529479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:40516)
0619 06:34:28.529695 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.529810 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:34:28.531351 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:28.530988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:35203 (local address 127.0.0.1:40522)
0619 06:34:28.531114 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:28.531202 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 06:34:28.657723 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 3940
I20260619 06:34:28.677976 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 3263
I20260619 06:34:28.679170 3438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:28.800086 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 3263
I20260619 06:34:28.821605 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 3362
I20260619 06:34:28.822746 3626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:28.942135 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 3362
2026-06-19T06:34:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9605 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:34:29 dist-test-slave-nf4n krb5kdc[4204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:34:29 dist-test-slave-nf4n krb5kdc[4204](info): set up 2 sockets
Jun 19 06:34:29 dist-test-slave-nf4n krb5kdc[4204](info): commencing operation
krb5kdc: starting...
W20260619 06:34:30.049713 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:34:30 dist-test-slave-nf4n krb5kdc[4204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.254.keytab.
Entry for principal oryx/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.254.keytab.
I20260619 06:34:30.266283 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:42471
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.254.keytab
--principal=oryx/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:42471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:30.592536 4218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:30.592810 4218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:30.592926 4218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:30.600451 4218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:30.600582 4218 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:34:30.600666 4218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:30.600765 4218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:30.600852 4218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:30.600950 4218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:30.611032 4218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:42471
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.254.keytab
--principal=oryx/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:42471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:30.612358 4218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:30.613808 4218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:30.625555 4223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:30.625768 4226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:30.625581 4224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:30.626477 4218 server_base.cc:1034] running on GCE node
I20260619 06:34:30.627219 4218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:30.628607 4218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:30.629818 4218 hybrid_clock.cc:648] HybridClock initialized: now 1781850870629743 us; error 56 us; skew 500 ppm
I20260619 06:34:30.630162 4218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:30 dist-test-slave-nf4n krb5kdc[4204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850870, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:30.633315 4218 init.cc:377] Logged in from keytab as oryx/127.28.104.254@KRBTEST.COM (short username oryx)
I20260619 06:34:30.634862 4218 webserver.cc:460] Webserver started at http://127.28.104.254:44967/ using document root <none> and password file <none>
I20260619 06:34:30.635583 4218 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:30.635712 4218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:30.636047 4218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:30.639114 4218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "9e279ab6c9f44ad186842249e1342f60"
format_stamp: "Formatted at 2026-06-19 06:34:30 on dist-test-slave-nf4n"
server_key: "4cab4765f99632036b9d281c90d667e5"
server_key_iv: "49939c542a697ccfb2269f08388678d6"
server_key_version: "encryptionkey@0"
I20260619 06:34:30.639912 4218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "9e279ab6c9f44ad186842249e1342f60"
format_stamp: "Formatted at 2026-06-19 06:34:30 on dist-test-slave-nf4n"
server_key: "4cab4765f99632036b9d281c90d667e5"
server_key_iv: "49939c542a697ccfb2269f08388678d6"
server_key_version: "encryptionkey@0"
I20260619 06:34:30.645116 4218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:34:30.648521 4233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:30.649928 4218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:30.650127 4218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "9e279ab6c9f44ad186842249e1342f60"
format_stamp: "Formatted at 2026-06-19 06:34:30 on dist-test-slave-nf4n"
server_key: "4cab4765f99632036b9d281c90d667e5"
server_key_iv: "49939c542a697ccfb2269f08388678d6"
server_key_version: "encryptionkey@0"
I20260619 06:34:30.650339 4218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:30.676894 4218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:30.681010 4218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:30.681318 4218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:30.703035 4218 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:42471
I20260619 06:34:30.703053 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:42471 every 8 connection(s)
I20260619 06:34:30.704443 4218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:30.707847 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 4218
I20260619 06:34:30.708081 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:30.708770 4295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:30.708911 29091 external_mini_cluster.cc:1428] Setting key 66816d4fd3bc182941b70236bafc4dcf
I20260619 06:34:30.724155 4295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60: Bootstrap starting.
Jun 19 06:34:30 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:30.727671 4295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:30.728732 4295 log.cc:824] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:30.732312 4295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60: No bootstrap required, opened a new log
I20260619 06:34:30.733698 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:30.710983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.28.104.254:42471)
0619 06:34:30.711797 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:30.711808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:30.711853 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:34:30.713700 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:30.713722 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:30.714223 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:30.714707 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:30.716606 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:30.717976 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:30.719725 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:30.720418 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:30.720570 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:30.725529 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:30.725537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:30.725552 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:30.725593 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:30.729269 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:30.730031 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:30.730042 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:30.730054 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:30.730283 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:30.730889 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:30.730893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:30.730895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:30.731391 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:30.731620 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:30.732154 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 06:34:30.732558 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":386,"threads_started":1}
I20260619 06:34:30.740602 4295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e279ab6c9f44ad186842249e1342f60" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42471 } }
I20260619 06:34:30.741058 4295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:30.741185 4295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e279ab6c9f44ad186842249e1342f60, State: Initialized, Role: FOLLOWER
I20260619 06:34:30.741760 4295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e279ab6c9f44ad186842249e1342f60" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42471 } }
I20260619 06:34:30.742003 4295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:30.742146 4295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:30.742316 4295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:30.744469 4295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e279ab6c9f44ad186842249e1342f60" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42471 } }
I20260619 06:34:30.745083 4295 leader_election.cc:302] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e279ab6c9f44ad186842249e1342f60; no voters:
I20260619 06:34:30.745720 4295 leader_election.cc:288] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:30.745936 4300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:30.747308 4300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [term 1 LEADER]: Becoming Leader. State: Replica: 9e279ab6c9f44ad186842249e1342f60, State: Running, Role: LEADER
I20260619 06:34:30.747941 4295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:30.747937 4300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e279ab6c9f44ad186842249e1342f60" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42471 } }
I20260619 06:34:30.751041 4302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e279ab6c9f44ad186842249e1342f60. Latest consensus state: current_term: 1 leader_uuid: "9e279ab6c9f44ad186842249e1342f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e279ab6c9f44ad186842249e1342f60" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42471 } } }
I20260619 06:34:30.751484 4302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:30.751497 4301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e279ab6c9f44ad186842249e1342f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e279ab6c9f44ad186842249e1342f60" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42471 } } }
I20260619 06:34:30.752032 4301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60 [sys.catalog]: This master's current role is: LEADER
W20260619 06:34:30.764484 4316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:34:30.764611 4316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:34:30.764776 4307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:30.767129 4307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:30.777230 4307 catalog_manager.cc:1347] Generated new cluster ID: 49761908f2ce4303bc11e1a2d4de5c53
I20260619 06:34:30.777370 4307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:30.806100 4307 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:30.807484 4307 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:30.814759 4307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60: Generated new TSK 0
I20260619 06:34:30.815894 4307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab.
Entry for principal oryx/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab.
I20260619 06:34:30.878610 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:42471
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab
--principal=oryx/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:31.210960 4321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:31.211256 4321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:31.211354 4321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:31.218695 4321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:31.218807 4321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:31.218890 4321 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:34:31.218969 4321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:34:31.228933 4321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab
--principal=oryx/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.104.254:42471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:31.230237 4321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:31.231645 4321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:31.246451 4326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:31.246438 4329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:31.246474 4327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:31.247479 4321 server_base.cc:1034] running on GCE node
I20260619 06:34:31.248183 4321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:31.249320 4321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:31.250592 4321 hybrid_clock.cc:648] HybridClock initialized: now 1781850871250515 us; error 42 us; skew 500 ppm
I20260619 06:34:31.250964 4321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:31 dist-test-slave-nf4n krb5kdc[4204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850871, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:31.258634 4321 init.cc:377] Logged in from keytab as oryx/127.28.104.193@KRBTEST.COM (short username oryx)
I20260619 06:34:31.260223 4321 webserver.cc:460] Webserver started at http://127.28.104.193:44335/ using document root <none> and password file <none>
I20260619 06:34:31.260983 4321 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:31.261157 4321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:31.261500 4321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:31.264503 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "96e96153039347ea87ccebe9338d2449"
format_stamp: "Formatted at 2026-06-19 06:34:31 on dist-test-slave-nf4n"
server_key: "54f0a637367157494242c8108d42a6b4"
server_key_iv: "046317186138a4d96b7ddd4b9d49816d"
server_key_version: "encryptionkey@0"
I20260619 06:34:31.265411 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "96e96153039347ea87ccebe9338d2449"
format_stamp: "Formatted at 2026-06-19 06:34:31 on dist-test-slave-nf4n"
server_key: "54f0a637367157494242c8108d42a6b4"
server_key_iv: "046317186138a4d96b7ddd4b9d49816d"
server_key_version: "encryptionkey@0"
I20260619 06:34:31.270562 4321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 06:34:31.274067 4336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:31.275422 4321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:34:31.275609 4321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "96e96153039347ea87ccebe9338d2449"
format_stamp: "Formatted at 2026-06-19 06:34:31 on dist-test-slave-nf4n"
server_key: "54f0a637367157494242c8108d42a6b4"
server_key_iv: "046317186138a4d96b7ddd4b9d49816d"
server_key_version: "encryptionkey@0"
I20260619 06:34:31.275825 4321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:31.295009 4321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:31.299593 4321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:31.299866 4321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:31.302639 4321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:31.302768 4321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:31.302938 4321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:31.303027 4321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:34:31 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850871, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:31.327987 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:31.303070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:34045 (local address 127.28.104.254:42471)
0619 06:34:31.303541 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:31.303548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:31.304801 (+ 1253us) server_negotiation.cc:407] Connection header received
0619 06:34:31.306809 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:31.306815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:31.306910 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:31.307025 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:31.309239 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.310351 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.312968 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.313667 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.313828 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:31.319339 (+ 5511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:31.319350 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:31.319358 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:31.319412 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:31.323722 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:31.324847 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:31.324852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:31.324856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:31.325050 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:31.325755 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:31.325759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:31.325761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:31.326150 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:31.326314 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:31.327517 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0619 06:34:31.327712 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":244,"threads_started":1}
I20260619 06:34:31.330338 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:31.303688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:42471 (local address 127.28.104.193:34045)
0619 06:34:31.304580 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:31.304632 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:31.306520 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:31.307267 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:31.307276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:31.308057 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:31.309013 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:31.309063 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.310509 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.310516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:31.312807 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:31.312821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.314041 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.314047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:31.314438 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:31.315377 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:31.315410 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:31.319023 (+ 3613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:31.323929 (+ 4906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:31.323936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:31.323950 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:31.324697 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:31.325174 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:31.325185 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:31.325192 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:31.325566 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:31.326329 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:31.326341 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:31.326985 (+ 644us) client_negotiation.cc:769] Sending connection context
0619 06:34:31.327377 (+ 392us) client_negotiation.cc:240] Negotiation successful
0619 06:34:31.328155 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":363,"threads_started":1}
I20260619 06:34:31.351243 4321 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:36277
I20260619 06:34:31.351253 4457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:36277 every 8 connection(s)
I20260619 06:34:31.352934 4321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:34:31.355491 4458 heartbeater.cc:347] Connected to a master server at 127.28.104.254:42471
I20260619 06:34:31.355751 4458 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:31.355897 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:31.339098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:42843 (local address 127.28.104.254:42471)
0619 06:34:31.339291 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:31.339300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:31.340287 (+ 987us) server_negotiation.cc:407] Connection header received
0619 06:34:31.340508 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:31.340516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:31.340585 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:31.340669 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:31.342132 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.342871 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.344975 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.345803 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.345954 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:31.348798 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:31.348808 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:31.348816 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:31.348871 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:31.352528 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:31.353347 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:31.353353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:31.353357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:31.353583 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:31.354143 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:31.354150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:31.354154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:31.354538 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:31.354703 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:31.355280 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 06:34:31.355556 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:34:31.355983 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:31.339455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:42471 (local address 127.28.104.193:42843)
0619 06:34:31.340151 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:31.340177 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:31.340362 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:31.340734 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:31.340742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:31.341541 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:31.341962 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:31.341975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.343106 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.343113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:31.344796 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:31.344817 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.346002 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.346008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:31.346177 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:31.346908 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:31.346943 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:31.348490 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:31.352678 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:31.352682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:31.352685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:31.353223 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:31.353730 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:31.353737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:31.353740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:31.353982 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:31.354785 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:31.354788 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:31.354984 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 06:34:31.355143 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 06:34:31.355484 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":333,"threads_started":1}
I20260619 06:34:31.356799 4458 heartbeater.cc:511] Master 127.28.104.254:42471 requested a full tablet report, sending...
I20260619 06:34:31.359031 4250 ts_manager.cc:194] Registered new tserver with Master: 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277)
I20260619 06:34:31.360716 4250 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:34045
I20260619 06:34:31.360879 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 4321
I20260619 06:34:31.361135 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:34:31.361846 29091 external_mini_cluster.cc:1428] Setting key 7eda8c1d1c5b7d636868e23aa7688c9e
WARNING: no policy specified for oryx/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab.
Entry for principal oryx/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab.
I20260619 06:34:31.408998 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:42471
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab
--principal=oryx/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:31.735769 4462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:31.736101 4462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:31.736279 4462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:31.743778 4462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:31.743935 4462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:31.744094 4462 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:34:31.744232 4462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:34:31.754277 4462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab
--principal=oryx/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.104.254:42471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:31.755657 4462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:31.757082 4462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:31.770020 4467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:31.770234 4468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:31.770586 4470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:31.770954 4462 server_base.cc:1034] running on GCE node
I20260619 06:34:31.771613 4462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:31.772533 4462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:31.773778 4462 hybrid_clock.cc:648] HybridClock initialized: now 1781850871773709 us; error 47 us; skew 500 ppm
I20260619 06:34:31.774084 4462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:31 dist-test-slave-nf4n krb5kdc[4204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850871, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:31.777434 4462 init.cc:377] Logged in from keytab as oryx/127.28.104.194@KRBTEST.COM (short username oryx)
I20260619 06:34:31.779031 4462 webserver.cc:460] Webserver started at http://127.28.104.194:33645/ using document root <none> and password file <none>
I20260619 06:34:31.779754 4462 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:31.779881 4462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:31.780259 4462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:31.783178 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "a1ea25a311ea49a1a269c44813055d54"
format_stamp: "Formatted at 2026-06-19 06:34:31 on dist-test-slave-nf4n"
server_key: "d9a38eb1f67c005d12c170bb1daee61b"
server_key_iv: "b036b33fcb3c51af243f741a106962b9"
server_key_version: "encryptionkey@0"
I20260619 06:34:31.783975 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "a1ea25a311ea49a1a269c44813055d54"
format_stamp: "Formatted at 2026-06-19 06:34:31 on dist-test-slave-nf4n"
server_key: "d9a38eb1f67c005d12c170bb1daee61b"
server_key_iv: "b036b33fcb3c51af243f741a106962b9"
server_key_version: "encryptionkey@0"
I20260619 06:34:31.789178 4462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 06:34:31.792701 4477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:31.794119 4462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:31.794298 4462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "a1ea25a311ea49a1a269c44813055d54"
format_stamp: "Formatted at 2026-06-19 06:34:31 on dist-test-slave-nf4n"
server_key: "d9a38eb1f67c005d12c170bb1daee61b"
server_key_iv: "b036b33fcb3c51af243f741a106962b9"
server_key_version: "encryptionkey@0"
I20260619 06:34:31.794513 4462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:31.807937 4462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:31.811915 4462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:31.812180 4462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:31.814980 4462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:31.815076 4462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:31.815227 4462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:31.815301 4462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:34:31 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850871, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:31.841801 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:31.815413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:45699 (local address 127.28.104.254:42471)
0619 06:34:31.815596 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:31.815604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:31.817217 (+ 1613us) server_negotiation.cc:407] Connection header received
0619 06:34:31.819456 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:31.819462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:31.819564 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:31.819677 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:31.822003 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.822809 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.826099 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.826638 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.826881 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:31.832896 (+ 6015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:31.832906 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:31.832914 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:31.832980 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:31.836845 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:31.838100 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:31.838107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:31.838112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:31.838271 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:31.838909 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:31.838915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:31.838919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:31.839214 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:31.839384 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:31.841238 (+ 1854us) server_negotiation.cc:299] Negotiation successful
0619 06:34:31.841492 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:34:31.842206 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:31.816124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:42471 (local address 127.28.104.194:45699)
0619 06:34:31.816975 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:31.817084 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:31.819112 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:31.819946 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:31.819960 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:31.820818 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:31.821734 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:31.821764 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.822987 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.822995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:31.825923 (+ 2928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:31.825939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.827053 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.827064 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:31.827548 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:31.828567 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:31.828617 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:31.832534 (+ 3917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:31.837081 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:31.837095 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:31.837121 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:31.837896 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:31.838418 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:31.838425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:31.838430 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:31.838763 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:31.839366 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:31.839374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:31.839850 (+ 476us) client_negotiation.cc:769] Sending connection context
0619 06:34:31.840146 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 06:34:31.840670 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":347,"threads_started":1}
I20260619 06:34:31.870002 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:31.855109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:60891 (local address 127.28.104.254:42471)
0619 06:34:31.855326 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:31.855334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:31.856225 (+ 891us) server_negotiation.cc:407] Connection header received
0619 06:34:31.856442 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:31.856447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:31.856523 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:31.856625 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:31.857726 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.858668 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.860138 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.860552 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.860683 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:31.863261 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:31.863271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:31.863279 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:31.863334 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:31.867143 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:31.867885 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:31.867894 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:31.867907 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:31.868052 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:31.868434 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:31.868440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:31.868443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:31.868770 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:31.868937 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:31.869420 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 06:34:31.869655 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:34:31.870128 4462 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:46137
I20260619 06:34:31.870275 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:31.855345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:42471 (local address 127.28.104.194:60891)
0619 06:34:31.856099 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:31.856117 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:31.856302 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:31.856650 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:31.856655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:31.857208 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:31.857579 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:31.857589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.858799 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.858805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:31.859933 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:31.859952 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:31.860749 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:31.860758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:31.860957 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:31.861725 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:31.861755 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:31.862995 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:31.867287 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:31.867292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:31.867296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:31.867758 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:31.868172 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:31.868178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:31.868182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:31.868315 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:31.868892 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:31.868897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:31.869156 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 06:34:31.869326 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 06:34:31.869735 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":402,"threads_started":1}
I20260619 06:34:31.870338 4597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:46137 every 8 connection(s)
I20260619 06:34:31.871807 4462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:34:31.873992 4598 heartbeater.cc:347] Connected to a master server at 127.28.104.254:42471
I20260619 06:34:31.874233 4598 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:31.875026 4598 heartbeater.cc:511] Master 127.28.104.254:42471 requested a full tablet report, sending...
I20260619 06:34:31.876369 4250 ts_manager.cc:194] Registered new tserver with Master: a1ea25a311ea49a1a269c44813055d54 (127.28.104.194:46137)
I20260619 06:34:31.877259 4250 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:45699
I20260619 06:34:31.880745 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 4462
I20260619 06:34:31.880896 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:34:31.881341 29091 external_mini_cluster.cc:1428] Setting key f389a49bdc562a7738eb5a913784cc31
WARNING: no policy specified for oryx/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab.
Entry for principal oryx/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab.
I20260619 06:34:31.926370 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:42471
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab
--principal=oryx/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:32.251936 4602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:32.252229 4602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:32.252326 4602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:32.259586 4602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:32.259693 4602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:32.259780 4602 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:34:32.259871 4602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:34:32.269609 4602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab
--principal=oryx/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.104.254:42471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:32.270872 4602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:32.272161 4602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:32.285277 4608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:32.285245 4611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:32.285279 4609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:32.286170 4602 server_base.cc:1034] running on GCE node
I20260619 06:34:32.286760 4602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:32.287703 4602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:32.288909 4602 hybrid_clock.cc:648] HybridClock initialized: now 1781850872288853 us; error 33 us; skew 500 ppm
I20260619 06:34:32.289319 4602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:32 dist-test-slave-nf4n krb5kdc[4204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850872, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:32.292538 4602 init.cc:377] Logged in from keytab as oryx/127.28.104.195@KRBTEST.COM (short username oryx)
I20260619 06:34:32.294163 4602 webserver.cc:460] Webserver started at http://127.28.104.195:43745/ using document root <none> and password file <none>
I20260619 06:34:32.294936 4602 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:32.295064 4602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:32.295434 4602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:32.298892 4602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "5703ff3d98544718a2c05ae846a2f8be"
format_stamp: "Formatted at 2026-06-19 06:34:32 on dist-test-slave-nf4n"
server_key: "7c2d00cf8a7988202041e65b19572e1d"
server_key_iv: "ae88d5873b6f5b220210f59989b03ecd"
server_key_version: "encryptionkey@0"
I20260619 06:34:32.299793 4602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "5703ff3d98544718a2c05ae846a2f8be"
format_stamp: "Formatted at 2026-06-19 06:34:32 on dist-test-slave-nf4n"
server_key: "7c2d00cf8a7988202041e65b19572e1d"
server_key_iv: "ae88d5873b6f5b220210f59989b03ecd"
server_key_version: "encryptionkey@0"
I20260619 06:34:32.305589 4602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260619 06:34:32.309365 4618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:32.310773 4602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:32.310974 4602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "5703ff3d98544718a2c05ae846a2f8be"
format_stamp: "Formatted at 2026-06-19 06:34:32 on dist-test-slave-nf4n"
server_key: "7c2d00cf8a7988202041e65b19572e1d"
server_key_iv: "ae88d5873b6f5b220210f59989b03ecd"
server_key_version: "encryptionkey@0"
I20260619 06:34:32.311224 4602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:32.330837 4602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:32.336170 4602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:32.336555 4602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:32.340063 4602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:32.340186 4602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:32.340334 4602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:32.340417 4602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:34:32 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850872, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:32.364110 4458 heartbeater.cc:503] Master 127.28.104.254:42471 was elected leader, sending a full tablet report...
I20260619 06:34:32.371531 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.340662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:33443 (local address 127.28.104.254:42471)
0619 06:34:32.340853 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:32.340860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:32.342633 (+ 1773us) server_negotiation.cc:407] Connection header received
0619 06:34:32.346144 (+ 3511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:32.346150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:32.346242 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:32.346365 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:32.348790 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.350303 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.354430 (+ 4127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.355367 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.355548 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.362696 (+ 7148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:32.362705 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:32.362711 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:32.362761 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:32.367234 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:32.368625 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:32.368632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:32.368635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:32.368776 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:32.369513 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:32.369522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:32.369527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:32.369930 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:32.370106 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:32.370924 (+ 818us) server_negotiation.cc:299] Negotiation successful
0619 06:34:32.371167 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:34:32.372609 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.341437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:42471 (local address 127.28.104.195:33443)
0619 06:34:32.342405 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:32.342476 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:32.345658 (+ 3182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:32.346700 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:32.346718 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:32.347612 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:32.348543 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.348580 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.350663 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.350669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.354168 (+ 3499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.354194 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.355999 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.356012 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.356441 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.357664 (+ 1223us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:32.357713 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:32.362283 (+ 4570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:32.367417 (+ 5134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:32.367431 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:32.367467 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:32.368437 (+ 970us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:32.368904 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:32.368913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:32.368920 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:32.369308 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:32.370081 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:32.370090 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:32.370482 (+ 392us) client_negotiation.cc:769] Sending connection context
0619 06:34:32.370785 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 06:34:32.371356 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":357,"threads_started":1}
I20260619 06:34:32.396613 4602 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:44817
I20260619 06:34:32.396662 4738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:44817 every 8 connection(s)
I20260619 06:34:32.398204 4602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:34:32.402086 4739 heartbeater.cc:347] Connected to a master server at 127.28.104.254:42471
I20260619 06:34:32.402428 4739 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:32.403424 4739 heartbeater.cc:511] Master 127.28.104.254:42471 requested a full tablet report, sending...
I20260619 06:34:32.405088 4250 ts_manager.cc:194] Registered new tserver with Master: 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817)
I20260619 06:34:32.405616 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.388524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:36987 (local address 127.28.104.254:42471)
0619 06:34:32.388784 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:32.388795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:32.389717 (+ 922us) server_negotiation.cc:407] Connection header received
0619 06:34:32.389860 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:32.389869 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:32.389997 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:32.390151 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:32.391107 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.392715 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.394886 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.395682 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.395848 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.398166 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:32.398174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:32.398178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:32.398226 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:32.402526 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:32.403422 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:32.403427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:32.403429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:32.403619 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:32.404049 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:32.404052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:32.404054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:32.404325 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:32.404520 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:32.404936 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 06:34:32.405234 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:34:32.405668 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.388710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:42471 (local address 127.28.104.195:36987)
0619 06:34:32.389604 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:32.389620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:32.389741 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:32.390160 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:32.390164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:32.390647 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:32.390940 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.390952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.392888 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.392891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.394739 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.394751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.395952 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.395961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.396198 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.397007 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:32.397078 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:32.397939 (+ 861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:32.402656 (+ 4717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:32.402661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:32.402665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:32.403286 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:32.403751 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:32.403756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:32.403760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:32.403932 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:32.404458 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:32.404463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:32.404661 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 06:34:32.404809 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 06:34:32.405190 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":507,"threads_started":1}
I20260619 06:34:32.406236 4250 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:33443
I20260619 06:34:32.408178 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 4602
I20260619 06:34:32.408390 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:34:32.408944 29091 external_mini_cluster.cc:1428] Setting key 56072ae5a053a20a0a6bcc71337d0437
I20260619 06:34:32.412083 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:34:32 dist-test-slave-nf4n krb5kdc[4204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:34:32 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:32 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:34:32.464439 4344 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:32.452998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.28.104.254:42471)
0619 06:34:32.453220 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:32.453224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:32.453592 (+ 368us) server_negotiation.cc:407] Connection header received
0619 06:34:32.453834 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:32.453838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:32.453905 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:32.454011 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:32.455622 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.456367 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.457850 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.458352 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.458463 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.464153 (+ 5690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38294: BlockingRecv error: recv got EOF from 127.0.0.1:38294 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Jun 19 06:34:32 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:32.481787 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.468211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.28.104.254:42471)
0619 06:34:32.468418 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:32.468423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:32.468745 (+ 322us) server_negotiation.cc:407] Connection header received
0619 06:34:32.468891 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:32.468895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:32.468995 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:32.469143 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:32.470242 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.471071 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.472267 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.472706 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.472804 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.476757 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:32.476762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:32.476765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:32.476807 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:32.479464 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:32.480125 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:32.480129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:32.480131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:32.480300 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:32.480619 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:32.480622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:32.480624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:32.480893 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:32.481191 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:32.481385 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 06:34:32.481557 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:34:32.486008 4250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:34:32.493211 4250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:34:32.514415 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.506410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:46137 (local address 127.0.0.1:37180)
0619 06:34:32.507081 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:32.507103 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:32.507212 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:32.508232 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:32.508237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:32.508265 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:32.508614 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.508623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.509995 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.510001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.512075 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.512087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.513429 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.513434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.513613 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.513662 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 06:34:32.513798 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 06:34:32.514009 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":363,"threads_started":1}
I20260619 06:34:32.515329 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.506857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.28.104.194:46137)
0619 06:34:32.507662 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:32.507671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:32.507697 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:34:32.507792 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:32.507799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:32.508051 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:32.508249 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:32.508769 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.509827 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.512243 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.513222 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.513420 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.514755 (+ 1335us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:32.514837 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 06:34:32.515014 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":903,"threads_started":1}
I20260619 06:34:32.516399 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.505230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:44817 (local address 127.0.0.1:36400)
0619 06:34:32.506043 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:32.506093 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:32.506285 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:32.507139 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:32.507147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:32.507183 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:32.507626 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.507640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.510462 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.510470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.513268 (+ 2798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.513287 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.515640 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.515646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.515867 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.515904 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:34:32.515981 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:34:32.516106 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":323,"threads_started":1}
I20260619 06:34:32.517426 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.505642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.28.104.195:44817)
0619 06:34:32.506404 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:32.506413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:32.506447 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:34:32.506564 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:32.506576 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:32.506885 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:32.507871 (+ 986us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:32.508251 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.510278 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.513527 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.515409 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.516699 (+ 1290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.516855 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:32.516915 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 06:34:32.517113 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":782,"threads_started":1}
I20260619 06:34:32.519771 4527 tablet_service.cc:1467] Processing CreateTablet for tablet 11df20b87e9c455bbf59f4c7458173f6 (DEFAULT_TABLE table=test-table [id=3f5bd3af1af543b09ef53359843c95f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:32.521278 4527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11df20b87e9c455bbf59f4c7458173f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:32.522149 4668 tablet_service.cc:1467] Processing CreateTablet for tablet 11df20b87e9c455bbf59f4c7458173f6 (DEFAULT_TABLE table=test-table [id=3f5bd3af1af543b09ef53359843c95f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:32.522898 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.507637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.28.104.193:36277)
0619 06:34:32.516057 (+ 8420us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:32.516065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:32.516098 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:34:32.516205 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:32.516220 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:32.516440 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:32.517119 (+ 679us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:32.517321 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.518808 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.520410 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.521866 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.522039 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.522201 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:32.522445 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 06:34:32.522604 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8241,"thread_start_us":8305,"threads_started":1}
I20260619 06:34:32.523447 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.507108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36277 (local address 127.0.0.1:60542)
0619 06:34:32.514981 (+ 7873us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:32.515044 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:32.515146 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:32.516684 (+ 1538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:32.516689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:32.516714 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:32.516901 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.516913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.518929 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.518933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.520227 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.520241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.522045 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.522051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.522251 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.522284 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:34:32.522389 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:34:32.522537 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7777,"thread_start_us":8331,"threads_started":1}
I20260619 06:34:32.523599 4668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11df20b87e9c455bbf59f4c7458173f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:32.527442 4388 tablet_service.cc:1467] Processing CreateTablet for tablet 11df20b87e9c455bbf59f4c7458173f6 (DEFAULT_TABLE table=test-table [id=3f5bd3af1af543b09ef53359843c95f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:32.528888 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11df20b87e9c455bbf59f4c7458173f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:32.534714 4763 tablet_bootstrap.cc:492] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: Bootstrap starting.
I20260619 06:34:32.538244 4763 tablet_bootstrap.cc:654] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:32.539251 4763 log.cc:824] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:32.541962 4764 tablet_bootstrap.cc:492] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: Bootstrap starting.
I20260619 06:34:32.542287 4763 tablet_bootstrap.cc:492] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: No bootstrap required, opened a new log
I20260619 06:34:32.542620 4763 ts_tablet_manager.cc:1397] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 06:34:32.546032 4764 tablet_bootstrap.cc:654] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:32.547039 4766 tablet_bootstrap.cc:492] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: Bootstrap starting.
I20260619 06:34:32.547425 4764 log.cc:824] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:32.550474 4764 tablet_bootstrap.cc:492] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: No bootstrap required, opened a new log
I20260619 06:34:32.550774 4764 ts_tablet_manager.cc:1397] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 06:34:32.550952 4766 tablet_bootstrap.cc:654] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:32.552249 4766 log.cc:824] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:32.554095 4763 raft_consensus.cc:348] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } }
I20260619 06:34:32.554553 4763 raft_consensus.cc:374] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:32.554656 4763 raft_consensus.cc:729] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5703ff3d98544718a2c05ae846a2f8be, State: Initialized, Role: FOLLOWER
I20260619 06:34:32.555605 4766 tablet_bootstrap.cc:492] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: No bootstrap required, opened a new log
I20260619 06:34:32.555415 4763 consensus_queue.cc:260] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } }
I20260619 06:34:32.556416 4766 ts_tablet_manager.cc:1397] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 06:34:32.557279 4763 ts_tablet_manager.cc:1428] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260619 06:34:32.557734 4739 heartbeater.cc:503] Master 127.28.104.254:42471 was elected leader, sending a full tablet report...
I20260619 06:34:32.560165 4764 raft_consensus.cc:348] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } }
I20260619 06:34:32.560534 4764 raft_consensus.cc:374] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:32.560639 4764 raft_consensus.cc:729] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ea25a311ea49a1a269c44813055d54, State: Initialized, Role: FOLLOWER
I20260619 06:34:32.561252 4764 consensus_queue.cc:260] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } }
I20260619 06:34:32.565430 4766 raft_consensus.cc:348] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } }
I20260619 06:34:32.565881 4766 raft_consensus.cc:374] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:32.566002 4766 raft_consensus.cc:729] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e96153039347ea87ccebe9338d2449, State: Initialized, Role: FOLLOWER
I20260619 06:34:32.566012 4598 heartbeater.cc:503] Master 127.28.104.254:42471 was elected leader, sending a full tablet report...
I20260619 06:34:32.566641 4764 ts_tablet_manager.cc:1428] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260619 06:34:32.566634 4766 consensus_queue.cc:260] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } }
I20260619 06:34:32.568197 4766 ts_tablet_manager.cc:1428] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260619 06:34:32.605677 4459 tablet.cc:2369] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:34:32.624085 4599 tablet.cc:2369] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:34:32.653440 4740 tablet.cc:2369] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:34:32.775305 4770 raft_consensus.cc:482] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:34:32.775741 4770 raft_consensus.cc:504] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } }
I20260619 06:34:32.778226 4770 leader_election.cc:288] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817), 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277)
I20260619 06:34:32.786558 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.779665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:57157 (local address 127.28.104.193:36277)
0619 06:34:32.779892 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:32.779899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:32.779947 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:34:32.780209 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:32.780218 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:32.780317 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:32.780385 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:32.781160 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.782220 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.784434 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.785609 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.785732 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.785785 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:32.786177 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 06:34:32.786273 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:34:32.786674 4773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.779203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36277 (local address 127.28.104.194:57157)
0619 06:34:32.779868 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:32.779885 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:32.780033 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:32.780616 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:32.780623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:32.780652 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:32.780994 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.781007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.782390 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.782395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.784277 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.784289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.785817 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.785823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.786000 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.786039 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:34:32.786144 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:34:32.786280 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":348,"threads_started":1}
I20260619 06:34:32.787384 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.778902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:33987 (local address 127.28.104.195:44817)
0619 06:34:32.779123 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:32.779130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:32.780131 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 06:34:32.780657 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:32.780665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:32.780749 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:32.780893 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:32.781750 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.782875 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.785327 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.786338 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.786493 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.786558 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:32.787007 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 06:34:32.787130 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:34:32.787436 4772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.778467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:44817 (local address 127.28.104.194:33987)
0619 06:34:32.779165 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:32.779187 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:32.780307 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:32.781132 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:32.781137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:32.781165 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:32.781570 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.781584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.783079 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.783090 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.785183 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:32.785200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.786577 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.786586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:32.786790 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.786866 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 06:34:32.786983 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 06:34:32.787111 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":300,"threads_started":1}
I20260619 06:34:32.787886 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11df20b87e9c455bbf59f4c7458173f6" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e96153039347ea87ccebe9338d2449" is_pre_election: true
I20260619 06:34:32.788393 4408 raft_consensus.cc:2454] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 0.
I20260619 06:34:32.789131 4479 leader_election.cc:302] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96e96153039347ea87ccebe9338d2449, a1ea25a311ea49a1a269c44813055d54; no voters:
I20260619 06:34:32.789569 4770 raft_consensus.cc:2781] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:34:32.789688 4770 raft_consensus.cc:482] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:34:32.789808 4770 raft_consensus.cc:3037] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:32.789710 4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11df20b87e9c455bbf59f4c7458173f6" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5703ff3d98544718a2c05ae846a2f8be" is_pre_election: true
I20260619 06:34:32.790244 4688 raft_consensus.cc:2454] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 0.
I20260619 06:34:32.792225 4770 raft_consensus.cc:504] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } }
I20260619 06:34:32.792783 4770 leader_election.cc:288] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 election: Requested vote from peers 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817), 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277)
I20260619 06:34:32.793257 4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11df20b87e9c455bbf59f4c7458173f6" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5703ff3d98544718a2c05ae846a2f8be"
I20260619 06:34:32.793342 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11df20b87e9c455bbf59f4c7458173f6" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e96153039347ea87ccebe9338d2449"
I20260619 06:34:32.793432 4688 raft_consensus.cc:3037] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:32.793588 4408 raft_consensus.cc:3037] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:32.795584 4688 raft_consensus.cc:2454] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 1.
I20260619 06:34:32.795939 4479 leader_election.cc:302] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5703ff3d98544718a2c05ae846a2f8be, a1ea25a311ea49a1a269c44813055d54; no voters:
I20260619 06:34:32.796048 4408 raft_consensus.cc:2454] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 1.
I20260619 06:34:32.796245 4770 raft_consensus.cc:2781] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:32.796732 4770 raft_consensus.cc:686] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 1 LEADER]: Becoming Leader. State: Replica: a1ea25a311ea49a1a269c44813055d54, State: Running, Role: LEADER
I20260619 06:34:32.797269 4770 consensus_queue.cc:237] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } }
I20260619 06:34:32.802016 4250 catalog_manager.cc:5495] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1ea25a311ea49a1a269c44813055d54 (127.28.104.194). New cstate: current_term: 1 leader_uuid: "a1ea25a311ea49a1a269c44813055d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } health_report { overall_health: UNKNOWN } } }
I20260619 06:34:32.884706 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:32.879532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.28.104.194:46137)
0619 06:34:32.879733 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:32.879738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:32.879872 (+ 134us) server_negotiation.cc:407] Connection header received
0619 06:34:32.880142 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:32.880147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:32.880257 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:32.880472 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:34:32.880789 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.881588 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.882805 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:32.883211 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:32.883301 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:32.883664 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:34:32.884146 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:34:32.884246 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:32.884322 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 06:34:32.884383 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 06:34:32.892925 4688 raft_consensus.cc:1261] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Refusing update from remote peer a1ea25a311ea49a1a269c44813055d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:32.892918 4408 raft_consensus.cc:1261] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Refusing update from remote peer a1ea25a311ea49a1a269c44813055d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:32.893931 4774 consensus_queue.cc:1035] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:32.894272 4770 consensus_queue.cc:1035] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:32.913156 4780 mvcc.cc:205] Tried to move back new op lower bound from 7298461175359299584 to 7298461174981513216. Current Snapshot: MvccSnapshot[applied={T|T < 7298461175359299584}]
I20260619 06:34:32.920758 4250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:32.921094 4250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:32.924993 4250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60: Sending DeleteTablet for 3 replicas of tablet 11df20b87e9c455bbf59f4c7458173f6
I20260619 06:34:32.926241 4668 tablet_service.cc:1514] Processing DeleteTablet for tablet 11df20b87e9c455bbf59f4c7458173f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:32 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:36400
I20260619 06:34:32.926366 4527 tablet_service.cc:1514] Processing DeleteTablet for tablet 11df20b87e9c455bbf59f4c7458173f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:32 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:37180
I20260619 06:34:32.926431 4388 tablet_service.cc:1514] Processing DeleteTablet for tablet 11df20b87e9c455bbf59f4c7458173f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:32 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:60542
I20260619 06:34:32.932729 4250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:34:32.934895 4250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:34:32.939188 4789 tablet_replica.cc:330] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: stopping tablet replica
I20260619 06:34:32.939683 4789 raft_consensus.cc:2229] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:32.940228 4789 raft_consensus.cc:2258] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:32.941140 4788 tablet_replica.cc:330] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: stopping tablet replica
I20260619 06:34:32.941864 4788 raft_consensus.cc:2229] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:34:32.942663 4788 raft_consensus.cc:2258] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:32.944605 4788 ts_tablet_manager.cc:1905] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:32.944605 4789 ts_tablet_manager.cc:1905] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:32.947824 4527 tablet_service.cc:1467] Processing CreateTablet for tablet 539e62aa811240cf8a39799c7dfbdce2 (DEFAULT_TABLE table=test-table [id=1970e48ba1804135a58b59561069e24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:32.948078 4388 tablet_service.cc:1467] Processing CreateTablet for tablet 539e62aa811240cf8a39799c7dfbdce2 (DEFAULT_TABLE table=test-table [id=1970e48ba1804135a58b59561069e24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:32.948143 4667 tablet_service.cc:1467] Processing CreateTablet for tablet 539e62aa811240cf8a39799c7dfbdce2 (DEFAULT_TABLE table=test-table [id=1970e48ba1804135a58b59561069e24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:32.948285 4527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 539e62aa811240cf8a39799c7dfbdce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:32.948510 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 539e62aa811240cf8a39799c7dfbdce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:32.948626 4667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 539e62aa811240cf8a39799c7dfbdce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:32.950001 4787 tablet_replica.cc:330] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: stopping tablet replica
I20260619 06:34:32.952874 4789 ts_tablet_manager.cc:1918] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:32.953572 4787 raft_consensus.cc:2229] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:32.954264 4787 raft_consensus.cc:2258] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:32.955018 4764 tablet_bootstrap.cc:492] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54: Bootstrap starting.
I20260619 06:34:32.955669 4763 tablet_bootstrap.cc:492] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be: Bootstrap starting.
I20260619 06:34:32.956969 4764 tablet_bootstrap.cc:654] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:32.957288 4763 tablet_bootstrap.cc:654] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:32.961176 4788 ts_tablet_manager.cc:1918] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:32.961405 4788 log.cc:1196] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/11df20b87e9c455bbf59f4c7458173f6
I20260619 06:34:32.962030 4787 ts_tablet_manager.cc:1905] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:32.962662 4788 ts_tablet_manager.cc:1939] T 11df20b87e9c455bbf59f4c7458173f6 P a1ea25a311ea49a1a269c44813055d54: Deleting consensus metadata
I20260619 06:34:32.965200 4235 catalog_manager.cc:4841] TS a1ea25a311ea49a1a269c44813055d54 (127.28.104.194:46137): tablet 11df20b87e9c455bbf59f4c7458173f6 (table test-table [id=3f5bd3af1af543b09ef53359843c95f7]) successfully deleted
I20260619 06:34:32.954924 4766 tablet_bootstrap.cc:492] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449: Bootstrap starting.
I20260619 06:34:32.967810 4766 tablet_bootstrap.cc:654] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:32.969177 4789 log.cc:1196] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/11df20b87e9c455bbf59f4c7458173f6
I20260619 06:34:32.970579 4766 tablet_bootstrap.cc:492] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449: No bootstrap required, opened a new log
I20260619 06:34:32.970758 4766 ts_tablet_manager.cc:1397] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260619 06:34:32.971755 4766 raft_consensus.cc:348] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:32.972105 4766 raft_consensus.cc:374] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:32.972203 4766 raft_consensus.cc:729] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e96153039347ea87ccebe9338d2449, State: Initialized, Role: FOLLOWER
I20260619 06:34:32.972667 4766 consensus_queue.cc:260] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:32.974426 4766 ts_tablet_manager.cc:1428] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260619 06:34:32.974872 4789 ts_tablet_manager.cc:1939] T 11df20b87e9c455bbf59f4c7458173f6 P 96e96153039347ea87ccebe9338d2449: Deleting consensus metadata
I20260619 06:34:32.980584 4235 catalog_manager.cc:4841] TS 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277): tablet 11df20b87e9c455bbf59f4c7458173f6 (table test-table [id=3f5bd3af1af543b09ef53359843c95f7]) successfully deleted
I20260619 06:34:32.981628 4764 tablet_bootstrap.cc:492] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54: No bootstrap required, opened a new log
I20260619 06:34:32.981899 4764 ts_tablet_manager.cc:1397] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.012s
I20260619 06:34:32.982692 4763 tablet_bootstrap.cc:492] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be: No bootstrap required, opened a new log
I20260619 06:34:32.982908 4763 ts_tablet_manager.cc:1397] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.016s
I20260619 06:34:32.983904 4763 raft_consensus.cc:348] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:32.984221 4763 raft_consensus.cc:374] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:32.984364 4763 raft_consensus.cc:729] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5703ff3d98544718a2c05ae846a2f8be, State: Initialized, Role: FOLLOWER
I20260619 06:34:32.984638 4764 raft_consensus.cc:348] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:32.984781 4763 consensus_queue.cc:260] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:32.984989 4764 raft_consensus.cc:374] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:32.985236 4764 raft_consensus.cc:729] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ea25a311ea49a1a269c44813055d54, State: Initialized, Role: FOLLOWER
I20260619 06:34:32.985621 4764 consensus_queue.cc:260] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:32.986588 4763 ts_tablet_manager.cc:1428] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 06:34:32.986905 4787 ts_tablet_manager.cc:1918] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:32.987069 4787 log.cc:1196] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/11df20b87e9c455bbf59f4c7458173f6
I20260619 06:34:32.987465 4764 ts_tablet_manager.cc:1428] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 06:34:32.987625 4787 ts_tablet_manager.cc:1939] T 11df20b87e9c455bbf59f4c7458173f6 P 5703ff3d98544718a2c05ae846a2f8be: Deleting consensus metadata
I20260619 06:34:32.991678 4235 catalog_manager.cc:4841] TS 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817): tablet 11df20b87e9c455bbf59f4c7458173f6 (table test-table [id=3f5bd3af1af543b09ef53359843c95f7]) successfully deleted
I20260619 06:34:33.189507 4774 raft_consensus.cc:482] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:34:33.189792 4774 raft_consensus.cc:504] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.190536 4774 leader_election.cc:288] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277), 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817)
I20260619 06:34:33.191322 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539e62aa811240cf8a39799c7dfbdce2" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e96153039347ea87ccebe9338d2449" is_pre_election: true
I20260619 06:34:33.191355 4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539e62aa811240cf8a39799c7dfbdce2" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5703ff3d98544718a2c05ae846a2f8be" is_pre_election: true
I20260619 06:34:33.191603 4408 raft_consensus.cc:2454] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 0.
I20260619 06:34:33.191643 4688 raft_consensus.cc:2454] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 0.
I20260619 06:34:33.192144 4479 leader_election.cc:302] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96e96153039347ea87ccebe9338d2449, a1ea25a311ea49a1a269c44813055d54; no voters:
I20260619 06:34:33.192513 4774 raft_consensus.cc:2781] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:34:33.192644 4774 raft_consensus.cc:482] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:34:33.192806 4774 raft_consensus.cc:3037] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:33.196087 4774 raft_consensus.cc:504] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.196790 4774 leader_election.cc:288] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 election: Requested vote from peers 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277), 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817)
I20260619 06:34:33.197342 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539e62aa811240cf8a39799c7dfbdce2" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e96153039347ea87ccebe9338d2449"
I20260619 06:34:33.197436 4688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539e62aa811240cf8a39799c7dfbdce2" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5703ff3d98544718a2c05ae846a2f8be"
I20260619 06:34:33.197580 4408 raft_consensus.cc:3037] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:33.197690 4688 raft_consensus.cc:3037] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:33.200073 4688 raft_consensus.cc:2454] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 1.
I20260619 06:34:33.200608 4408 raft_consensus.cc:2454] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 1.
I20260619 06:34:33.200609 4479 leader_election.cc:302] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5703ff3d98544718a2c05ae846a2f8be, a1ea25a311ea49a1a269c44813055d54; no voters:
I20260619 06:34:33.200943 4774 raft_consensus.cc:2781] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:33.201149 4774 raft_consensus.cc:686] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 1 LEADER]: Becoming Leader. State: Replica: a1ea25a311ea49a1a269c44813055d54, State: Running, Role: LEADER
I20260619 06:34:33.201764 4774 consensus_queue.cc:237] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.204924 4250 catalog_manager.cc:5495] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1ea25a311ea49a1a269c44813055d54 (127.28.104.194). New cstate: current_term: 1 leader_uuid: "a1ea25a311ea49a1a269c44813055d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } health_report { overall_health: UNKNOWN } } }
Jun 19 06:34:33 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850870, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.254@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:33.248651 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.231707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.28.104.254:42471)
0619 06:34:33.232933 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:33.232938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:33.232960 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:34:33.233010 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:33.233014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:33.233150 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:33.233251 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:33.234875 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.235732 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.237644 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.238079 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.238203 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.242498 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:33.242503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:33.242506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:33.242534 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:33.245660 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:33.246530 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:33.246533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:33.246536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:33.246654 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:33.247055 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:33.247058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:33.247060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:33.247309 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:33.247424 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:33.247958 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 06:34:33.248348 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":1092,"thread_start_us":865,"threads_started":1}
I20260619 06:34:33.248692 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.231742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:42471 (local address 127.0.0.1:38298)
0619 06:34:33.232455 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:33.232486 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:33.232642 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:33.233325 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:33.233332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:33.234354 (+ 1022us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:33.234669 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.234684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.235892 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.235898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.237479 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.237497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.238275 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.238283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.238454 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.239260 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:33.239283 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:33.242273 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:33.245812 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:33.245825 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:33.245834 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:33.246406 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:33.246764 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:33.246770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:33.246774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:33.246949 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:33.247450 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:33.247459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:33.247678 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 06:34:33.247874 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 06:34:33.248201 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":352,"threads_started":1}
I20260619 06:34:33.254415 4250 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:38298:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:34:33.255582 4250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:34:33.262694 4388 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8909b4ee7241c3bbddce966038f8ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d9d46f6dcc3448fb401ee1876d76ee9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:34:33.262969 4527 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8909b4ee7241c3bbddce966038f8ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d9d46f6dcc3448fb401ee1876d76ee9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:34:33.263300 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8909b4ee7241c3bbddce966038f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:33.263257 4667 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8909b4ee7241c3bbddce966038f8ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d9d46f6dcc3448fb401ee1876d76ee9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:34:33.263588 4527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8909b4ee7241c3bbddce966038f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:33.263773 4667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8909b4ee7241c3bbddce966038f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:33.270766 4766 tablet_bootstrap.cc:492] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: Bootstrap starting.
I20260619 06:34:33.270766 4763 tablet_bootstrap.cc:492] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: Bootstrap starting.
I20260619 06:34:33.272480 4766 tablet_bootstrap.cc:654] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:33.272480 4763 tablet_bootstrap.cc:654] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:33.273329 4764 tablet_bootstrap.cc:492] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: Bootstrap starting.
I20260619 06:34:33.275213 4764 tablet_bootstrap.cc:654] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:33.284490 4764 tablet_bootstrap.cc:492] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: No bootstrap required, opened a new log
I20260619 06:34:33.284780 4764 ts_tablet_manager.cc:1397] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260619 06:34:33.285089 4763 tablet_bootstrap.cc:492] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: No bootstrap required, opened a new log
I20260619 06:34:33.285308 4763 ts_tablet_manager.cc:1397] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260619 06:34:33.285926 4764 raft_consensus.cc:348] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.286267 4764 raft_consensus.cc:374] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:33.286242 4763 raft_consensus.cc:348] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.286403 4764 raft_consensus.cc:729] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ea25a311ea49a1a269c44813055d54, State: Initialized, Role: FOLLOWER
I20260619 06:34:33.286511 4763 raft_consensus.cc:374] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:33.286590 4763 raft_consensus.cc:729] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5703ff3d98544718a2c05ae846a2f8be, State: Initialized, Role: FOLLOWER
I20260619 06:34:33.286871 4763 consensus_queue.cc:260] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.287189 4764 consensus_queue.cc:260] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.287357 4769 tablet_replica.cc:439] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } }
I20260619 06:34:33.287765 4769 tablet_replica.cc:442] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:34:33.287818 4774 tablet_replica.cc:439] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } }
I20260619 06:34:33.288075 4774 tablet_replica.cc:442] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:34:33.289541 4763 ts_tablet_manager.cc:1428] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:34:33.291412 4766 tablet_bootstrap.cc:492] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: No bootstrap required, opened a new log
I20260619 06:34:33.291616 4766 ts_tablet_manager.cc:1397] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.016s
I20260619 06:34:33.292243 4764 ts_tablet_manager.cc:1428] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260619 06:34:33.292397 4766 raft_consensus.cc:348] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.292662 4766 raft_consensus.cc:374] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:33.292749 4766 raft_consensus.cc:729] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e96153039347ea87ccebe9338d2449, State: Initialized, Role: FOLLOWER
I20260619 06:34:33.293001 4766 consensus_queue.cc:260] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.293489 4771 tablet_replica.cc:439] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } }
I20260619 06:34:33.293802 4771 tablet_replica.cc:442] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:34:33.294878 4766 ts_tablet_manager.cc:1428] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 06:34:33.501224 4769 raft_consensus.cc:482] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:34:33.501468 4769 raft_consensus.cc:504] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.502897 4769 leader_election.cc:288] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277), a1ea25a311ea49a1a269c44813055d54 (127.28.104.194:46137)
I20260619 06:34:33.509831 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.503149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36277 (local address 127.28.104.195:53203)
0619 06:34:33.503595 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:33.503616 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:33.503743 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:33.504280 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:33.504286 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:33.504318 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:33.504611 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.504621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.505960 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.505964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.507834 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.507850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.509201 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.509205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.509330 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.509354 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:34:33.509426 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 06:34:33.509530 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":242,"threads_started":1}
I20260619 06:34:33.509935 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.503271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:53203 (local address 127.28.104.193:36277)
0619 06:34:33.503754 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:33.503760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:33.503783 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:34:33.503900 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:33.503905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:33.503985 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:33.504102 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:33.504765 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.505820 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.508001 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.509057 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.509230 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.509311 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:33.509478 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 06:34:33.509618 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":268,"threads_started":1}
I20260619 06:34:33.510607 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.503972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:33499 (local address 127.28.104.194:46137)
0619 06:34:33.504667 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:33.504694 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:33.504715 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:34:33.504780 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:33.504785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:33.504868 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:33.504981 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:34:33.505480 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.506315 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.508380 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.509610 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.509763 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.509834 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:33.510192 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 06:34:33.510321 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":500,"threads_started":1}
I20260619 06:34:33.510540 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8909b4ee7241c3bbddce966038f8ba" candidate_uuid: "5703ff3d98544718a2c05ae846a2f8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e96153039347ea87ccebe9338d2449" is_pre_election: true
I20260619 06:34:33.510641 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.503804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:46137 (local address 127.28.104.195:33499)
0619 06:34:33.504303 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:33.504317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:33.504434 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:33.505095 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:33.505099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:33.505119 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:34:33.505337 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.505345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.506440 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.506445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.508241 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.508251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.509812 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.509824 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.510050 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.510087 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:34:33.510166 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:34:33.510292 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":1,"spinlock_wait_cycles":3968,"thread_start_us":266,"threads_started":1}
I20260619 06:34:33.510818 4408 raft_consensus.cc:2454] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5703ff3d98544718a2c05ae846a2f8be in term 0.
I20260619 06:34:33.511299 4620 leader_election.cc:302] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5703ff3d98544718a2c05ae846a2f8be, 96e96153039347ea87ccebe9338d2449; no voters:
I20260619 06:34:33.511427 4547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8909b4ee7241c3bbddce966038f8ba" candidate_uuid: "5703ff3d98544718a2c05ae846a2f8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ea25a311ea49a1a269c44813055d54" is_pre_election: true
I20260619 06:34:33.511704 4769 raft_consensus.cc:2781] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:34:33.511844 4769 raft_consensus.cc:482] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:34:33.511814 4547 raft_consensus.cc:2454] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5703ff3d98544718a2c05ae846a2f8be in term 0.
I20260619 06:34:33.511952 4769 raft_consensus.cc:3037] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:33.514461 4769 raft_consensus.cc:504] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.515138 4769 leader_election.cc:288] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [CANDIDATE]: Term 1 election: Requested vote from peers 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277), a1ea25a311ea49a1a269c44813055d54 (127.28.104.194:46137)
I20260619 06:34:33.515681 4547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8909b4ee7241c3bbddce966038f8ba" candidate_uuid: "5703ff3d98544718a2c05ae846a2f8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ea25a311ea49a1a269c44813055d54"
I20260619 06:34:33.515697 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8909b4ee7241c3bbddce966038f8ba" candidate_uuid: "5703ff3d98544718a2c05ae846a2f8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e96153039347ea87ccebe9338d2449"
I20260619 06:34:33.515888 4547 raft_consensus.cc:3037] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:33.515944 4408 raft_consensus.cc:3037] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:33.518853 4408 raft_consensus.cc:2454] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5703ff3d98544718a2c05ae846a2f8be in term 1.
I20260619 06:34:33.518867 4547 raft_consensus.cc:2454] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5703ff3d98544718a2c05ae846a2f8be in term 1.
I20260619 06:34:33.519232 4620 leader_election.cc:302] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5703ff3d98544718a2c05ae846a2f8be, 96e96153039347ea87ccebe9338d2449; no voters:
I20260619 06:34:33.519546 4769 raft_consensus.cc:2781] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:33.519923 4769 raft_consensus.cc:686] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [term 1 LEADER]: Becoming Leader. State: Replica: 5703ff3d98544718a2c05ae846a2f8be, State: Running, Role: LEADER
I20260619 06:34:33.520287 4769 consensus_queue.cc:237] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.522208 4808 tablet_replica.cc:439] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: TxnStatusTablet state changed. Reason: New leader 5703ff3d98544718a2c05ae846a2f8be. Latest consensus state: current_term: 1 leader_uuid: "5703ff3d98544718a2c05ae846a2f8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } }
I20260619 06:34:33.522626 4808 tablet_replica.cc:442] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:34:33.523455 4810 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:34:33.523676 4810 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:34:33.523949 4250 catalog_manager.cc:5495] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be reported cstate change: term changed from 0 to 1, leader changed from <none> to 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195). New cstate: current_term: 1 leader_uuid: "5703ff3d98544718a2c05ae846a2f8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } health_report { overall_health: HEALTHY } } }
I20260619 06:34:33.587529 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.580554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:44817 (local address 127.0.0.1:36404)
0619 06:34:33.580786 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:33.580809 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:33.580967 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:33.581922 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:33.581925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:33.581934 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:34:33.582081 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.582087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.583565 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.583568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.584582 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.584593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.585629 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.585633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.585809 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.585864 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:34:33.587042 (+ 1178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:34:33.587057 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 06:34:33.587110 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 06:34:33.587231 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 06:34:33.587746 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.580765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.28.104.195:44817)
0619 06:34:33.581464 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:33.581471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:33.581502 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:34:33.581608 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:33.581613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:33.581730 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:33.581899 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:34:33.582202 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.583439 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.584713 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.585493 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.585612 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.586087 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:34:33.586925 (+ 838us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:34:33.587098 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:33.587161 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 06:34:33.587277 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":407,"threads_started":1}
I20260619 06:34:33.592458 4408 raft_consensus.cc:1261] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Refusing update from remote peer 5703ff3d98544718a2c05ae846a2f8be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:33.592597 4547 raft_consensus.cc:1261] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Refusing update from remote peer 5703ff3d98544718a2c05ae846a2f8be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:33.593281 4808 consensus_queue.cc:1035] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:33.593645 4769 consensus_queue.cc:1035] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:33.597081 4771 tablet_replica.cc:439] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: TxnStatusTablet state changed. Reason: New leader 5703ff3d98544718a2c05ae846a2f8be. Latest consensus state: current_term: 1 leader_uuid: "5703ff3d98544718a2c05ae846a2f8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } }
I20260619 06:34:33.597381 4771 tablet_replica.cc:442] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:34:33.601395 4808 tablet_replica.cc:439] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5703ff3d98544718a2c05ae846a2f8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } }
I20260619 06:34:33.601711 4808 tablet_replica.cc:442] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:34:33.604998 4814 mvcc.cc:205] Tried to move back new op lower bound from 7298461178226610176 to 7298461177943126016. Current Snapshot: MvccSnapshot[applied={T|T < 7298461178226610176 or (T in {7298461178226610176})}]
I20260619 06:34:33.611429 4771 tablet_replica.cc:439] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5703ff3d98544718a2c05ae846a2f8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } }
I20260619 06:34:33.611655 4771 tablet_replica.cc:442] T 6e8909b4ee7241c3bbddce966038f8ba P 96e96153039347ea87ccebe9338d2449: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:34:33.613955 4774 tablet_replica.cc:439] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: TxnStatusTablet state changed. Reason: New leader 5703ff3d98544718a2c05ae846a2f8be. Latest consensus state: current_term: 1 leader_uuid: "5703ff3d98544718a2c05ae846a2f8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } }
I20260619 06:34:33.614264 4774 tablet_replica.cc:442] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:34:33.615932 4808 tablet_replica.cc:439] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5703ff3d98544718a2c05ae846a2f8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } }
I20260619 06:34:33.616201 4808 tablet_replica.cc:442] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:34:33.627038 4774 tablet_replica.cc:439] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5703ff3d98544718a2c05ae846a2f8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } } }
I20260619 06:34:33.627295 4774 tablet_replica.cc:442] T 6e8909b4ee7241c3bbddce966038f8ba P a1ea25a311ea49a1a269c44813055d54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:34:33.631542 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.624948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:53927 (local address 127.28.104.195:44817)
0619 06:34:33.625181 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:33.625190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:33.625631 (+ 441us) server_negotiation.cc:407] Connection header received
0619 06:34:33.625837 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:33.625844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:33.625912 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:33.626018 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:34:33.626460 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.627493 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.629661 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.630184 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.630301 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.630812 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:34:33.630911 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:34:33.631007 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:33.631146 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 06:34:33.631223 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:34:33.631757 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.624871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:44817 (local address 127.28.104.194:53927)
0619 06:34:33.625529 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:33.625548 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:33.625709 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:33.626032 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:33.626036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:33.626048 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:34:33.626317 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.626325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.627907 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.627912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.629533 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.629543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.630447 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.630452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.630584 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.630635 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:34:33.631019 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:34:33.631031 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 06:34:33.631125 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:34:33.631253 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":343,"threads_started":1}
I20260619 06:34:33.640810 4688 raft_consensus.cc:1261] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Refusing update from remote peer a1ea25a311ea49a1a269c44813055d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:33.640815 4408 raft_consensus.cc:1261] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Refusing update from remote peer a1ea25a311ea49a1a269c44813055d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:33.641342 4774 consensus_queue.cc:1035] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:33.641613 4770 consensus_queue.cc:1035] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:33.647439 4816 mvcc.cc:205] Tried to move back new op lower bound from 7298461178428248064 to 7298461176637657088. Current Snapshot: MvccSnapshot[applied={T|T < 7298461178428248064}]
I20260619 06:34:33.687958 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.673340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:35003 (local address 127.28.104.194:46137)
0619 06:34:33.673577 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:33.673587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:33.681734 (+ 8147us) server_negotiation.cc:407] Connection header received
0619 06:34:33.682042 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:33.682050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:33.682134 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:33.682194 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:34:33.682975 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.683684 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.685908 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.686431 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.686581 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.687229 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:34:33.687361 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:34:33.687465 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:33.687582 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 06:34:33.687674 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:34:33.688006 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.673279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:46137 (local address 127.28.104.195:35003)
0619 06:34:33.681610 (+ 8331us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:33.681639 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:33.681862 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:33.682407 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:33.682412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:33.682424 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:34:33.682795 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.682807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.683813 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.683819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.685776 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:33.685791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.686745 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.686752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:33.686955 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.687045 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:34:33.687471 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:34:33.687480 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:34:33.687539 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 06:34:33.687690 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8067,"thread_start_us":7927,"threads_started":1}
I20260619 06:34:33.722923 4248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:33.723197 4248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:33.726276 4248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60: Sending DeleteTablet for 3 replicas of tablet 539e62aa811240cf8a39799c7dfbdce2
I20260619 06:34:33.727048 4388 tablet_service.cc:1514] Processing DeleteTablet for tablet 539e62aa811240cf8a39799c7dfbdce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:33 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:60542
I20260619 06:34:33.727043 4527 tablet_service.cc:1514] Processing DeleteTablet for tablet 539e62aa811240cf8a39799c7dfbdce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:33 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:37180
I20260619 06:34:33.727170 4667 tablet_service.cc:1514] Processing DeleteTablet for tablet 539e62aa811240cf8a39799c7dfbdce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:33 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:36400
I20260619 06:34:33.746778 4834 tablet_replica.cc:330] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be: stopping tablet replica
I20260619 06:34:33.747037 4834 raft_consensus.cc:2229] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:33.747352 4834 raft_consensus.cc:2258] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:33.750077 4834 ts_tablet_manager.cc:1905] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:33.754127 4833 tablet_replica.cc:330] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449: stopping tablet replica
I20260619 06:34:33.754671 4833 raft_consensus.cc:2229] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:33.755148 4833 raft_consensus.cc:2258] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:33.756750 4833 ts_tablet_manager.cc:1905] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:33.759657 4832 tablet_replica.cc:330] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54: stopping tablet replica
I20260619 06:34:33.759886 4832 raft_consensus.cc:2229] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:34:33.760277 4832 raft_consensus.cc:2258] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:33.764416 4833 ts_tablet_manager.cc:1918] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:34:33.764572 4833 log.cc:1196] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/539e62aa811240cf8a39799c7dfbdce2
I20260619 06:34:33.765087 4833 ts_tablet_manager.cc:1939] T 539e62aa811240cf8a39799c7dfbdce2 P 96e96153039347ea87ccebe9338d2449: Deleting consensus metadata
I20260619 06:34:33.765563 4832 ts_tablet_manager.cc:1905] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:33.765990 4834 ts_tablet_manager.cc:1918] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:34:33.766134 4834 log.cc:1196] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/539e62aa811240cf8a39799c7dfbdce2
I20260619 06:34:33.766256 4235 catalog_manager.cc:4841] TS 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277): tablet 539e62aa811240cf8a39799c7dfbdce2 (table test-table [id=1970e48ba1804135a58b59561069e24a]) successfully deleted
I20260619 06:34:33.766642 4834 ts_tablet_manager.cc:1939] T 539e62aa811240cf8a39799c7dfbdce2 P 5703ff3d98544718a2c05ae846a2f8be: Deleting consensus metadata
I20260619 06:34:33.767690 4235 catalog_manager.cc:4841] TS 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817): tablet 539e62aa811240cf8a39799c7dfbdce2 (table test-table [id=1970e48ba1804135a58b59561069e24a]) successfully deleted
I20260619 06:34:33.771414 4832 ts_tablet_manager.cc:1918] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:34:33.771564 4832 log.cc:1196] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/539e62aa811240cf8a39799c7dfbdce2
I20260619 06:34:33.771953 4832 ts_tablet_manager.cc:1939] T 539e62aa811240cf8a39799c7dfbdce2 P a1ea25a311ea49a1a269c44813055d54: Deleting consensus metadata
I20260619 06:34:33.772835 4235 catalog_manager.cc:4841] TS a1ea25a311ea49a1a269c44813055d54 (127.28.104.194:46137): tablet 539e62aa811240cf8a39799c7dfbdce2 (table test-table [id=1970e48ba1804135a58b59561069e24a]) successfully deleted
Jun 19 06:34:33 dist-test-slave-nf4n krb5kdc[4204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:34:33 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:33 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:34:33.820254 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:33.809584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.28.104.254:42471)
0619 06:34:33.810122 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:33.810128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:33.810395 (+ 267us) server_negotiation.cc:407] Connection header received
0619 06:34:33.810610 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:33.810614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:33.810676 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:33.810789 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:33.812081 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.812896 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.814208 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.814638 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.814793 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.819658 (+ 4865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38312: BlockingRecv error: recv got EOF from 127.0.0.1:38312 (error 108)
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":269,"threads_started":1}
Jun 19 06:34:33 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM
I20260619 06:34:33.839407 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:33.822862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.28.104.254:42471)
0619 06:34:33.823039 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:33.823044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:33.823319 (+ 275us) server_negotiation.cc:407] Connection header received
0619 06:34:33.823531 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:33.823534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:33.823593 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:33.823672 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:33.825557 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.826753 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.828035 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:33.828747 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:33.828902 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:33.832800 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:33.832808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:33.832814 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:33.832861 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:33.836557 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:33.837258 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:33.837265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:33.837269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:33.837452 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:33.837822 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:33.837828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:33.837832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:33.838220 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:33.838522 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:33.838717 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 06:34:33.838977 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:34:33.843293 4248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:34:33.843997 4248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:34:33.850980 4388 tablet_service.cc:1467] Processing CreateTablet for tablet 59b910e85f5546b9b47ba986a5ce0bb6 (DEFAULT_TABLE table=test-table [id=f61e2de853c94ac9979ad9eb82999058]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:33.851421 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b910e85f5546b9b47ba986a5ce0bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:33.856614 4848 tablet_bootstrap.cc:492] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449: Bootstrap starting.
I20260619 06:34:33.857156 4527 tablet_service.cc:1467] Processing CreateTablet for tablet 59b910e85f5546b9b47ba986a5ce0bb6 (DEFAULT_TABLE table=test-table [id=f61e2de853c94ac9979ad9eb82999058]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:33.857161 4667 tablet_service.cc:1467] Processing CreateTablet for tablet 59b910e85f5546b9b47ba986a5ce0bb6 (DEFAULT_TABLE table=test-table [id=f61e2de853c94ac9979ad9eb82999058]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:33.857587 4527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b910e85f5546b9b47ba986a5ce0bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:33.857646 4667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b910e85f5546b9b47ba986a5ce0bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:33.858773 4848 tablet_bootstrap.cc:654] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:33.870930 4850 tablet_bootstrap.cc:492] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be: Bootstrap starting.
I20260619 06:34:33.873963 4851 tablet_bootstrap.cc:492] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54: Bootstrap starting.
I20260619 06:34:33.874584 4850 tablet_bootstrap.cc:654] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:33.878019 4851 tablet_bootstrap.cc:654] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:33.883913 4851 tablet_bootstrap.cc:492] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54: No bootstrap required, opened a new log
I20260619 06:34:33.884116 4851 ts_tablet_manager.cc:1397] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260619 06:34:33.886281 4848 tablet_bootstrap.cc:492] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449: No bootstrap required, opened a new log
I20260619 06:34:33.886530 4848 ts_tablet_manager.cc:1397] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.008s
I20260619 06:34:33.886667 4851 raft_consensus.cc:348] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.886970 4851 raft_consensus.cc:374] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:33.887102 4851 raft_consensus.cc:729] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ea25a311ea49a1a269c44813055d54, State: Initialized, Role: FOLLOWER
I20260619 06:34:33.887557 4848 raft_consensus.cc:348] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.887904 4848 raft_consensus.cc:374] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:33.888038 4848 raft_consensus.cc:729] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e96153039347ea87ccebe9338d2449, State: Initialized, Role: FOLLOWER
I20260619 06:34:33.888350 4848 consensus_queue.cc:260] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.888726 4851 consensus_queue.cc:260] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.889220 4848 ts_tablet_manager.cc:1428] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 06:34:33.889395 4851 ts_tablet_manager.cc:1428] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260619 06:34:33.891448 4850 tablet_bootstrap.cc:492] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be: No bootstrap required, opened a new log
I20260619 06:34:33.891785 4850 ts_tablet_manager.cc:1397] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.005s
I20260619 06:34:33.892690 4850 raft_consensus.cc:348] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.892966 4850 raft_consensus.cc:374] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:33.893110 4850 raft_consensus.cc:729] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5703ff3d98544718a2c05ae846a2f8be, State: Initialized, Role: FOLLOWER
I20260619 06:34:33.893355 4850 consensus_queue.cc:260] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:33.894019 4850 ts_tablet_manager.cc:1428] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:34:34.124043 4774 raft_consensus.cc:482] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:34:34.124285 4774 raft_consensus.cc:504] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.124991 4774 leader_election.cc:288] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277), 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817)
I20260619 06:34:34.125586 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b910e85f5546b9b47ba986a5ce0bb6" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e96153039347ea87ccebe9338d2449" is_pre_election: true
I20260619 06:34:34.125599 4687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b910e85f5546b9b47ba986a5ce0bb6" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5703ff3d98544718a2c05ae846a2f8be" is_pre_election: true
I20260619 06:34:34.125841 4408 raft_consensus.cc:2454] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 0.
I20260619 06:34:34.125845 4687 raft_consensus.cc:2454] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 0.
I20260619 06:34:34.126340 4479 leader_election.cc:302] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5703ff3d98544718a2c05ae846a2f8be, a1ea25a311ea49a1a269c44813055d54; no voters:
I20260619 06:34:34.126657 4774 raft_consensus.cc:2781] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:34:34.126740 4774 raft_consensus.cc:482] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:34:34.126856 4774 raft_consensus.cc:3037] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:34.128955 4774 raft_consensus.cc:504] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.129474 4774 leader_election.cc:288] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 election: Requested vote from peers 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277), 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817)
I20260619 06:34:34.129953 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b910e85f5546b9b47ba986a5ce0bb6" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e96153039347ea87ccebe9338d2449"
I20260619 06:34:34.130036 4687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b910e85f5546b9b47ba986a5ce0bb6" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5703ff3d98544718a2c05ae846a2f8be"
I20260619 06:34:34.130167 4408 raft_consensus.cc:3037] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:34.130214 4687 raft_consensus.cc:3037] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:34.133138 4408 raft_consensus.cc:2454] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 1.
I20260619 06:34:34.133129 4687 raft_consensus.cc:2454] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 1.
I20260619 06:34:34.133517 4479 leader_election.cc:302] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96e96153039347ea87ccebe9338d2449, a1ea25a311ea49a1a269c44813055d54; no voters:
I20260619 06:34:34.133795 4774 raft_consensus.cc:2781] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:34.134016 4774 raft_consensus.cc:686] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 1 LEADER]: Becoming Leader. State: Replica: a1ea25a311ea49a1a269c44813055d54, State: Running, Role: LEADER
I20260619 06:34:34.134318 4774 consensus_queue.cc:237] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.136734 4249 catalog_manager.cc:5495] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1ea25a311ea49a1a269c44813055d54 (127.28.104.194). New cstate: current_term: 1 leader_uuid: "a1ea25a311ea49a1a269c44813055d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } health_report { overall_health: UNKNOWN } } }
I20260619 06:34:34.167537 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:34.163326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.28.104.194:46137)
0619 06:34:34.163526 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:34.163530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:34.163550 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:34:34.163696 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:34.163699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:34.163763 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:34.163846 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:34:34.164140 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:34.164818 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:34.166139 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:34.166529 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:34.166621 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:34.166972 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:34:34.167071 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:34:34.167134 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:34.167263 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 06:34:34.167328 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:34:34.170128 4687 raft_consensus.cc:1261] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Refusing update from remote peer a1ea25a311ea49a1a269c44813055d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:34.170288 4408 raft_consensus.cc:1261] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Refusing update from remote peer a1ea25a311ea49a1a269c44813055d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:34.170794 4831 consensus_queue.cc:1035] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:34.171082 4774 consensus_queue.cc:1035] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:34.181924 4249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:34.182128 4249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:34.184716 4249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60: Sending DeleteTablet for 3 replicas of tablet 59b910e85f5546b9b47ba986a5ce0bb6
I20260619 06:34:34.185552 4388 tablet_service.cc:1514] Processing DeleteTablet for tablet 59b910e85f5546b9b47ba986a5ce0bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:34 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:60542
I20260619 06:34:34.185652 4527 tablet_service.cc:1514] Processing DeleteTablet for tablet 59b910e85f5546b9b47ba986a5ce0bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:34 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:37180
I20260619 06:34:34.185708 4667 tablet_service.cc:1514] Processing DeleteTablet for tablet 59b910e85f5546b9b47ba986a5ce0bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:34 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:36400
I20260619 06:34:34.185952 4832 tablet_replica.cc:330] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54: stopping tablet replica
I20260619 06:34:34.185966 4834 tablet_replica.cc:330] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be: stopping tablet replica
I20260619 06:34:34.185989 4833 tablet_replica.cc:330] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449: stopping tablet replica
I20260619 06:34:34.186162 4832 raft_consensus.cc:2229] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:34:34.186167 4833 raft_consensus.cc:2229] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:34.186167 4834 raft_consensus.cc:2229] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:34.186374 4833 raft_consensus.cc:2258] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:34.186374 4834 raft_consensus.cc:2258] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:34.186482 4832 raft_consensus.cc:2258] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:34.187526 4834 ts_tablet_manager.cc:1905] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:34.187779 4832 ts_tablet_manager.cc:1905] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:34.187988 4833 ts_tablet_manager.cc:1905] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:34.187780 4249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:34:34.188683 4249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:34:34.192946 4833 ts_tablet_manager.cc:1918] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:34.193148 4833 log.cc:1196] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/59b910e85f5546b9b47ba986a5ce0bb6
I20260619 06:34:34.193639 4833 ts_tablet_manager.cc:1939] T 59b910e85f5546b9b47ba986a5ce0bb6 P 96e96153039347ea87ccebe9338d2449: Deleting consensus metadata
I20260619 06:34:34.194088 4834 ts_tablet_manager.cc:1918] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:34.194247 4834 log.cc:1196] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/59b910e85f5546b9b47ba986a5ce0bb6
I20260619 06:34:34.194679 4834 ts_tablet_manager.cc:1939] T 59b910e85f5546b9b47ba986a5ce0bb6 P 5703ff3d98544718a2c05ae846a2f8be: Deleting consensus metadata
I20260619 06:34:34.194941 4235 catalog_manager.cc:4841] TS 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277): tablet 59b910e85f5546b9b47ba986a5ce0bb6 (table test-table [id=f61e2de853c94ac9979ad9eb82999058]) successfully deleted
I20260619 06:34:34.195180 4832 ts_tablet_manager.cc:1918] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:34:34.195353 4832 log.cc:1196] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/59b910e85f5546b9b47ba986a5ce0bb6
I20260619 06:34:34.195801 4832 ts_tablet_manager.cc:1939] T 59b910e85f5546b9b47ba986a5ce0bb6 P a1ea25a311ea49a1a269c44813055d54: Deleting consensus metadata
I20260619 06:34:34.196461 4235 catalog_manager.cc:4841] TS 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817): tablet 59b910e85f5546b9b47ba986a5ce0bb6 (table test-table [id=f61e2de853c94ac9979ad9eb82999058]) successfully deleted
I20260619 06:34:34.196880 4235 catalog_manager.cc:4841] TS a1ea25a311ea49a1a269c44813055d54 (127.28.104.194:46137): tablet 59b910e85f5546b9b47ba986a5ce0bb6 (table test-table [id=f61e2de853c94ac9979ad9eb82999058]) successfully deleted
I20260619 06:34:34.199674 4527 tablet_service.cc:1467] Processing CreateTablet for tablet d6b823b8227b40159beda225bdffaee4 (DEFAULT_TABLE table=test-table [id=ea642c13548747e3be7f336f04ba7d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:34.199949 4667 tablet_service.cc:1467] Processing CreateTablet for tablet d6b823b8227b40159beda225bdffaee4 (DEFAULT_TABLE table=test-table [id=ea642c13548747e3be7f336f04ba7d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:34.200101 4527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b823b8227b40159beda225bdffaee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:34.200335 4667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b823b8227b40159beda225bdffaee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:34.204738 4851 tablet_bootstrap.cc:492] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54: Bootstrap starting.
I20260619 06:34:34.204836 4850 tablet_bootstrap.cc:492] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be: Bootstrap starting.
I20260619 06:34:34.206377 4850 tablet_bootstrap.cc:654] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:34.206431 4851 tablet_bootstrap.cc:654] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:34.207398 4388 tablet_service.cc:1467] Processing CreateTablet for tablet d6b823b8227b40159beda225bdffaee4 (DEFAULT_TABLE table=test-table [id=ea642c13548747e3be7f336f04ba7d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:34:34.207831 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b823b8227b40159beda225bdffaee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:34.208420 4851 tablet_bootstrap.cc:492] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54: No bootstrap required, opened a new log
I20260619 06:34:34.208581 4851 ts_tablet_manager.cc:1397] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 06:34:34.209501 4851 raft_consensus.cc:348] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.209759 4851 raft_consensus.cc:374] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:34.209853 4851 raft_consensus.cc:729] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ea25a311ea49a1a269c44813055d54, State: Initialized, Role: FOLLOWER
I20260619 06:34:34.210089 4851 consensus_queue.cc:260] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.210589 4851 ts_tablet_manager.cc:1428] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:34:34.212988 4848 tablet_bootstrap.cc:492] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449: Bootstrap starting.
I20260619 06:34:34.213991 4850 tablet_bootstrap.cc:492] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be: No bootstrap required, opened a new log
I20260619 06:34:34.214138 4850 ts_tablet_manager.cc:1397] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260619 06:34:34.214663 4848 tablet_bootstrap.cc:654] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:34.214900 4850 raft_consensus.cc:348] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.215162 4850 raft_consensus.cc:374] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:34.215245 4850 raft_consensus.cc:729] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5703ff3d98544718a2c05ae846a2f8be, State: Initialized, Role: FOLLOWER
I20260619 06:34:34.215485 4850 consensus_queue.cc:260] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.216682 4848 tablet_bootstrap.cc:492] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449: No bootstrap required, opened a new log
I20260619 06:34:34.216671 4850 ts_tablet_manager.cc:1428] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 06:34:34.216830 4848 ts_tablet_manager.cc:1397] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 06:34:34.217475 4848 raft_consensus.cc:348] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.217698 4848 raft_consensus.cc:374] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:34.217797 4848 raft_consensus.cc:729] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e96153039347ea87ccebe9338d2449, State: Initialized, Role: FOLLOWER
I20260619 06:34:34.218029 4848 consensus_queue.cc:260] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.218677 4848 ts_tablet_manager.cc:1428] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:34:34.261379 4774 raft_consensus.cc:482] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:34:34.261583 4774 raft_consensus.cc:504] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.262176 4774 leader_election.cc:288] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277), 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817)
I20260619 06:34:34.262758 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b823b8227b40159beda225bdffaee4" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e96153039347ea87ccebe9338d2449" is_pre_election: true
I20260619 06:34:34.262812 4687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b823b8227b40159beda225bdffaee4" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5703ff3d98544718a2c05ae846a2f8be" is_pre_election: true
I20260619 06:34:34.263013 4408 raft_consensus.cc:2454] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 0.
I20260619 06:34:34.263118 4687 raft_consensus.cc:2454] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 0.
I20260619 06:34:34.263489 4479 leader_election.cc:302] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96e96153039347ea87ccebe9338d2449, a1ea25a311ea49a1a269c44813055d54; no voters:
I20260619 06:34:34.263808 4774 raft_consensus.cc:2781] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:34:34.263926 4774 raft_consensus.cc:482] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:34:34.264039 4774 raft_consensus.cc:3037] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:34.266271 4774 raft_consensus.cc:504] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.266803 4774 leader_election.cc:288] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 election: Requested vote from peers 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277), 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817)
I20260619 06:34:34.267355 4687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b823b8227b40159beda225bdffaee4" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5703ff3d98544718a2c05ae846a2f8be"
I20260619 06:34:34.267323 4408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b823b8227b40159beda225bdffaee4" candidate_uuid: "a1ea25a311ea49a1a269c44813055d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e96153039347ea87ccebe9338d2449"
I20260619 06:34:34.267561 4687 raft_consensus.cc:3037] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:34.267580 4408 raft_consensus.cc:3037] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:34.269565 4687 raft_consensus.cc:2454] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 1.
I20260619 06:34:34.269651 4408 raft_consensus.cc:2454] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ea25a311ea49a1a269c44813055d54 in term 1.
I20260619 06:34:34.270010 4479 leader_election.cc:302] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5703ff3d98544718a2c05ae846a2f8be, a1ea25a311ea49a1a269c44813055d54; no voters:
I20260619 06:34:34.270246 4774 raft_consensus.cc:2781] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:34.270414 4774 raft_consensus.cc:686] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 1 LEADER]: Becoming Leader. State: Replica: a1ea25a311ea49a1a269c44813055d54, State: Running, Role: LEADER
I20260619 06:34:34.270686 4774 consensus_queue.cc:237] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } }
I20260619 06:34:34.273335 4248 catalog_manager.cc:5495] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1ea25a311ea49a1a269c44813055d54 (127.28.104.194). New cstate: current_term: 1 leader_uuid: "a1ea25a311ea49a1a269c44813055d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1ea25a311ea49a1a269c44813055d54" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 } health_report { overall_health: UNKNOWN } } }
I20260619 06:34:34.300061 4408 raft_consensus.cc:1261] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Refusing update from remote peer a1ea25a311ea49a1a269c44813055d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:34.300119 4687 raft_consensus.cc:1261] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Refusing update from remote peer a1ea25a311ea49a1a269c44813055d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:34:34.300634 4831 consensus_queue.cc:1035] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5703ff3d98544718a2c05ae846a2f8be" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:34.301050 4774 consensus_queue.cc:1035] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e96153039347ea87ccebe9338d2449" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:34:34.347868 4250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:34.348165 4250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:34:34.351161 4250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e279ab6c9f44ad186842249e1342f60: Sending DeleteTablet for 3 replicas of tablet d6b823b8227b40159beda225bdffaee4
I20260619 06:34:34.351994 4388 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b823b8227b40159beda225bdffaee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:34 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:60542
I20260619 06:34:34.352090 4527 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b823b8227b40159beda225bdffaee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:34 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:37180
I20260619 06:34:34.352288 4833 tablet_replica.cc:330] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449: stopping tablet replica
I20260619 06:34:34.352439 4832 tablet_replica.cc:330] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54: stopping tablet replica
I20260619 06:34:34.352421 4667 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b823b8227b40159beda225bdffaee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:34:34 UTC) from {username='oryx', principal='oryx/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:36400
I20260619 06:34:34.352490 4833 raft_consensus.cc:2229] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:34.352622 4832 raft_consensus.cc:2229] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:34:34.352692 4833 raft_consensus.cc:2258] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:34.352705 4834 tablet_replica.cc:330] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be: stopping tablet replica
I20260619 06:34:34.352855 4834 raft_consensus.cc:2229] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:34:34.352967 4832 raft_consensus.cc:2258] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:34.353113 4834 raft_consensus.cc:2258] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:34:34.354399 4833 ts_tablet_manager.cc:1905] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:34.354442 4834 ts_tablet_manager.cc:1905] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:34.354799 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 4321
I20260619 06:34:34.356222 4832 ts_tablet_manager.cc:1905] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:34:34.361521 4834 ts_tablet_manager.cc:1918] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:34:34.361670 4834 log.cc:1196] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/d6b823b8227b40159beda225bdffaee4
I20260619 06:34:34.362141 4834 ts_tablet_manager.cc:1939] T d6b823b8227b40159beda225bdffaee4 P 5703ff3d98544718a2c05ae846a2f8be: Deleting consensus metadata
I20260619 06:34:34.363127 4833 ts_tablet_manager.cc:1918] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:34:34.363253 4833 log.cc:1196] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/d6b823b8227b40159beda225bdffaee4
I20260619 06:34:34.363328 4235 catalog_manager.cc:4841] TS 5703ff3d98544718a2c05ae846a2f8be (127.28.104.195:44817): tablet d6b823b8227b40159beda225bdffaee4 (table test-table [id=ea642c13548747e3be7f336f04ba7d5e]) successfully deleted
I20260619 06:34:34.363687 4833 ts_tablet_manager.cc:1939] T d6b823b8227b40159beda225bdffaee4 P 96e96153039347ea87ccebe9338d2449: Deleting consensus metadata
I20260619 06:34:34.364773 4235 catalog_manager.cc:4841] TS 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277): tablet d6b823b8227b40159beda225bdffaee4 (table test-table [id=ea642c13548747e3be7f336f04ba7d5e]) successfully deleted
I20260619 06:34:34.366369 4832 ts_tablet_manager.cc:1918] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:34:34.366531 4832 log.cc:1196] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/d6b823b8227b40159beda225bdffaee4
I20260619 06:34:34.366902 4832 ts_tablet_manager.cc:1939] T d6b823b8227b40159beda225bdffaee4 P a1ea25a311ea49a1a269c44813055d54: Deleting consensus metadata
I20260619 06:34:34.367842 4235 catalog_manager.cc:4841] TS a1ea25a311ea49a1a269c44813055d54 (127.28.104.194:46137): tablet d6b823b8227b40159beda225bdffaee4 (table test-table [id=ea642c13548747e3be7f336f04ba7d5e]) successfully deleted
Jun 19 06:34:34 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.193@KRBTEST.COM
I20260619 06:34:34.389169 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:34.358472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.28.104.193:36277)
0619 06:34:34.373520 (+ 15048us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:34.373526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:34.373545 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:34:34.373597 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:34.373603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:34.373675 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:34.373782 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:34.375035 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:34.376238 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:34.377678 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:34.378377 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:34.378501 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:34.382638 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:34.382654 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:34.382675 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:34.382713 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:34.386363 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:34.387040 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:34.387050 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:34.387054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:34.387237 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:34.387567 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:34.387572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:34.387576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:34.388022 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:34.388292 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:34.388552 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 06:34:34.388781 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14925,"thread_start_us":14825,"threads_started":1}
I20260619 06:34:34.389853 4453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:34.522943 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 4321
I20260619 06:34:34.541723 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 4462
Jun 19 06:34:34 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.194@KRBTEST.COM
I20260619 06:34:34.560462 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:34.542621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.28.104.194:46137)
0619 06:34:34.542846 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:34.542855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:34.542893 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:34:34.542977 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:34.542985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:34.543086 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:34.543208 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:34.544460 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:34.545827 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:34.547105 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:34.547909 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:34.548056 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:34.552482 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:34.552508 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:34.552534 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:34.552600 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:34.557262 (+ 4662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:34.558022 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:34.558035 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:34.558042 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:34.558231 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:34.558579 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:34.558587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:34.558592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:34.559126 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:34.559489 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:34.559704 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 06:34:34.560000 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:34:34.561332 4593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:34.685870 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 4462
I20260619 06:34:34.710232 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 4602
I20260619 06:34:34.715133 4873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:34.713514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36277 (local address 127.28.104.195:59889)
0619 06:34:34.714727 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:34.714877 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1102,"thread_start_us":971,"threads_started":1}
W20260619 06:34:34.715478 4620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36277: connect: Connection refused (error 111)
I20260619 06:34:34.717275 4873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:34.716714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36277 (local address 127.28.104.195:43299)
0619 06:34:34.716887 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:34.716980 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:34:34.717545 4620 consensus_peers.cc:487] T 6e8909b4ee7241c3bbddce966038f8ba P 5703ff3d98544718a2c05ae846a2f8be -> Peer 96e96153039347ea87ccebe9338d2449 (127.28.104.193:36277): Couldn't send request to peer 96e96153039347ea87ccebe9338d2449. Status: Network error: Client connection negotiation failed: client connection to 127.28.104.193:36277: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Jun 19 06:34:34 dist-test-slave-nf4n krb5kdc[4204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.104.195@KRBTEST.COM
I20260619 06:34:34.727080 4872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:34.711076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.28.104.195:44817)
0619 06:34:34.711592 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:34.711597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:34.711611 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:34:34.711663 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:34.711667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:34.711722 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:34.711850 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:34.713337 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:34.714241 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:34.716149 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:34.716700 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:34.716788 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:34.721448 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:34.721459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:34.721476 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:34.721509 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:34.724213 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:34.724892 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:34.724900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:34.724902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:34.725189 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:34.725565 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:34.725573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:34.725577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:34.725927 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:34.726239 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:34.726522 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 06:34:34.726737 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":261,"threads_started":1}
I20260619 06:34:34.727862 4734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:34.855798 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 4602
I20260619 06:34:34.877637 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 4218
I20260619 06:34:34.878859 4290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:34.996505 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 4218
2026-06-19T06:34:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6053 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:34:35 dist-test-slave-nf4n krb5kdc[4879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:34:35 dist-test-slave-nf4n krb5kdc[4879](info): set up 2 sockets
Jun 19 06:34:35 dist-test-slave-nf4n krb5kdc[4879](info): commencing operation
krb5kdc: starting...
W20260619 06:34:36.105932 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:34:36 dist-test-slave-nf4n krb5kdc[4879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:34:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:34:36.332232 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:40197
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:35285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:40197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:36.669864 4892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:36.670243 4892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:36.670404 4892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:36.677851 4892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:36.678025 4892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:36.678140 4892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:36.678246 4892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:36.678337 4892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:36.688416 4892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:35285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:40197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:40197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:36.689819 4892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:36.691210 4892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:36.703430 4900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:36.703598 4897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:36.703536 4898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:36.704229 4892 server_base.cc:1034] running on GCE node
I20260619 06:34:36.705149 4892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:36.706578 4892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:36.707815 4892 hybrid_clock.cc:648] HybridClock initialized: now 1781850876707771 us; error 58 us; skew 500 ppm
I20260619 06:34:36.708192 4892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T06:34:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1793 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:34:36 dist-test-slave-nf4n krb5kdc[4906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:34:36 dist-test-slave-nf4n krb5kdc[4906](info): set up 2 sockets
Jun 19 06:34:36 dist-test-slave-nf4n krb5kdc[4906](info): commencing operation
krb5kdc: starting...
W20260619 06:34:37.902189 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:34:38 dist-test-slave-nf4n krb5kdc[4906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:34:38.132403 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:37635
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:37635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:38.457692 4919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:38.458060 4919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:38.458215 4919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:38.465544 4919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:38.465685 4919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:38.465793 4919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:38.465896 4919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:38.465974 4919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:38.476788 4919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:37635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:37635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:38.478184 4919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:38.479489 4919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:38.491827 4924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:38.491895 4925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:38.491803 4928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:38.492909 4919 server_base.cc:1034] running on GCE node
I20260619 06:34:38.493795 4919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:38.495273 4919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:38.496477 4919 hybrid_clock.cc:648] HybridClock initialized: now 1781850878496431 us; error 36 us; skew 500 ppm
I20260619 06:34:38.496834 4919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:38 dist-test-slave-nf4n krb5kdc[4906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850878, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:38.499989 4919 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:34:38.501452 4919 webserver.cc:460] Webserver started at http://127.28.104.254:46815/ using document root <none> and password file <none>
I20260619 06:34:38.502112 4919 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:38.502209 4919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:38.502542 4919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:38.505601 4919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "29b8aa98065c410f8a1534013a35b550"
format_stamp: "Formatted at 2026-06-19 06:34:38 on dist-test-slave-nf4n"
server_key: "15402af5bd2046cbdd6c65bc6a1a9c55"
server_key_iv: "ebbc989c57dd3fce8a98acfcdf268f52"
server_key_version: "encryptionkey@0"
I20260619 06:34:38.506392 4919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "29b8aa98065c410f8a1534013a35b550"
format_stamp: "Formatted at 2026-06-19 06:34:38 on dist-test-slave-nf4n"
server_key: "15402af5bd2046cbdd6c65bc6a1a9c55"
server_key_iv: "ebbc989c57dd3fce8a98acfcdf268f52"
server_key_version: "encryptionkey@0"
I20260619 06:34:38.511560 4919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:34:38.515023 4935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:38.516325 4919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:34:38.516518 4919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "29b8aa98065c410f8a1534013a35b550"
format_stamp: "Formatted at 2026-06-19 06:34:38 on dist-test-slave-nf4n"
server_key: "15402af5bd2046cbdd6c65bc6a1a9c55"
server_key_iv: "ebbc989c57dd3fce8a98acfcdf268f52"
server_key_version: "encryptionkey@0"
I20260619 06:34:38.516726 4919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:38.544783 4919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:38.548785 4919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:38.549113 4919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:38.567710 4919 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:37635
I20260619 06:34:38.567734 4986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:37635 every 8 connection(s)
I20260619 06:34:38.569180 4919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:38.573161 4987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:38.573884 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 4919
I20260619 06:34:38.574142 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:38.574791 29091 external_mini_cluster.cc:1428] Setting key 3f6a00df970a6ce1f7464f964030b67f
I20260619 06:34:38.587536 4987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550: Bootstrap starting.
Jun 19 06:34:38 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:38.590831 4987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:38.592233 4987 log.cc:824] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:38.595747 4987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550: No bootstrap required, opened a new log
I20260619 06:34:38.599432 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:38.576917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.28.104.254:37635)
0619 06:34:38.577852 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:38.577868 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:38.577923 (+ 55us) server_negotiation.cc:407] Connection header received
0619 06:34:38.579873 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:38.579902 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:38.580439 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:38.580905 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:38.582331 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:38.584291 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:38.585564 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:38.586562 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:38.586735 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:38.590751 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:38.590763 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:38.590782 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:38.590837 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:38.594623 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:38.595318 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:38.595326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:38.595334 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:38.595556 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:38.596054 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:38.596059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:38.596062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:38.596798 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:38.597121 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:38.597558 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 06:34:38.598030 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":380,"threads_started":1}
I20260619 06:34:38.605427 4987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } }
I20260619 06:34:38.605777 4987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:38.605850 4987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b8aa98065c410f8a1534013a35b550, State: Initialized, Role: FOLLOWER
I20260619 06:34:38.606330 4987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } }
I20260619 06:34:38.606525 4987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:38.606619 4987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:38.606746 4987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:38.609207 4987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } }
I20260619 06:34:38.609660 4987 leader_election.cc:302] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b8aa98065c410f8a1534013a35b550; no voters:
I20260619 06:34:38.610172 4987 leader_election.cc:288] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:38.610325 4992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:38.611065 4992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 1 LEADER]: Becoming Leader. State: Replica: 29b8aa98065c410f8a1534013a35b550, State: Running, Role: LEADER
I20260619 06:34:38.611675 4992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } }
I20260619 06:34:38.612380 4987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:38.614593 4994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8aa98065c410f8a1534013a35b550. Latest consensus state: current_term: 1 leader_uuid: "29b8aa98065c410f8a1534013a35b550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } } }
I20260619 06:34:38.615011 4994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:38.615144 4993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29b8aa98065c410f8a1534013a35b550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } } }
I20260619 06:34:38.616135 4993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:38.616649 5000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:38.622337 5000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:38.630944 5000 catalog_manager.cc:1347] Generated new cluster ID: 3f1c81ea5faf47c1884eb1f6cee70686
I20260619 06:34:38.631090 5000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:38.660239 5000 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:38.662276 5000 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:38.671008 5000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550: Generated new TSK 0
I20260619 06:34:38.672154 5000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:34:38.738895 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37635
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:39.079286 5013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:39.079589 5013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:39.079772 5013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:39.086977 5013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:39.087081 5013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:39.087280 5013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:34:39.097155 5013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:39.098510 5013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:39.099900 5013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:39.112998 5018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:39.113065 5021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:39.113088 5019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:39.114248 5013 server_base.cc:1034] running on GCE node
I20260619 06:34:39.114828 5013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:39.115757 5013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:39.117004 5013 hybrid_clock.cc:648] HybridClock initialized: now 1781850879116928 us; error 61 us; skew 500 ppm
I20260619 06:34:39.117426 5013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:39 dist-test-slave-nf4n krb5kdc[4906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850879, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:39.121318 5013 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:34:39.122958 5013 webserver.cc:460] Webserver started at http://127.28.104.193:34475/ using document root <none> and password file <none>
I20260619 06:34:39.123692 5013 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:39.123852 5013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:39.124221 5013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:39.127378 5013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "f65b2ee1ae6a48cca42758cb7854ae92"
format_stamp: "Formatted at 2026-06-19 06:34:39 on dist-test-slave-nf4n"
server_key: "bd4d6617930bb46cae0c792490413614"
server_key_iv: "0be4e2345e2b75ab07746308411f229e"
server_key_version: "encryptionkey@0"
I20260619 06:34:39.128190 5013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "f65b2ee1ae6a48cca42758cb7854ae92"
format_stamp: "Formatted at 2026-06-19 06:34:39 on dist-test-slave-nf4n"
server_key: "bd4d6617930bb46cae0c792490413614"
server_key_iv: "0be4e2345e2b75ab07746308411f229e"
server_key_version: "encryptionkey@0"
I20260619 06:34:39.133466 5013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:34:39.136927 5028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:39.138325 5013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:34:39.138499 5013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "f65b2ee1ae6a48cca42758cb7854ae92"
format_stamp: "Formatted at 2026-06-19 06:34:39 on dist-test-slave-nf4n"
server_key: "bd4d6617930bb46cae0c792490413614"
server_key_iv: "0be4e2345e2b75ab07746308411f229e"
server_key_version: "encryptionkey@0"
I20260619 06:34:39.138690 5013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:39.164270 5013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:39.168494 5013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:39.168788 5013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:39.169886 5013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:39.171258 5013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:39.171329 5013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:39.171491 5013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:39.171550 5013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:39.204371 5013 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:37651
I20260619 06:34:39.204509 5141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:37651 every 8 connection(s)
I20260619 06:34:39.205766 5013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:34:39.213621 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 5013
I20260619 06:34:39.213887 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:34:39.214505 29091 external_mini_cluster.cc:1428] Setting key 97674c3db9219e468426530eba6b1c3e
Jun 19 06:34:39 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850879, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:39.228091 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:39.208421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:36541 (local address 127.28.104.254:37635)
0619 06:34:39.208822 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:39.208826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:39.209953 (+ 1127us) server_negotiation.cc:407] Connection header received
0619 06:34:39.211792 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:39.211797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:39.211867 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:39.211971 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:39.213781 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:39.214574 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:39.216357 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:39.216757 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:39.216902 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:39.221285 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:39.221291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:39.221294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:39.221328 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:39.224747 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:39.225549 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:39.225558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:39.225565 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:39.225712 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:39.226116 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:39.226124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:39.226141 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:39.226510 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:39.226675 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:39.227357 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 06:34:39.227607 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":195,"threads_started":1}
I20260619 06:34:39.228761 5145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:39.209074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:36541)
0619 06:34:39.209784 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:39.209824 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:39.211507 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:39.212240 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:39.212249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:39.212843 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:39.213572 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:39.213612 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:39.214788 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:39.214794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:39.216191 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:39.216204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:39.216989 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:39.216998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:39.217265 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:39.218001 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:39.218024 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:39.220967 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:39.224899 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:39.224907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:39.224921 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:39.225416 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:39.225818 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:39.225821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:39.225823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:39.226001 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:39.226633 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:39.226638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:39.226956 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 06:34:39.227185 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 06:34:39.227605 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":270,"threads_started":1}
I20260619 06:34:39.230410 5142 heartbeater.cc:347] Connected to a master server at 127.28.104.254:37635
I20260619 06:34:39.230760 5142 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:39.231484 5142 heartbeater.cc:511] Master 127.28.104.254:37635 requested a full tablet report, sending...
I20260619 06:34:39.233770 4952 ts_manager.cc:194] Registered new tserver with Master: f65b2ee1ae6a48cca42758cb7854ae92 (127.28.104.193:37651)
I20260619 06:34:39.235224 4952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:36541
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:34:39.266078 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37635
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:39.593406 5148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:39.593684 5148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:39.593838 5148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:39.601686 5148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:39.601792 5148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:39.601958 5148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:34:39.611671 5148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:39.613137 5148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:39.614939 5148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:39.628473 5153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:39.628720 5156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:39.628604 5154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:39.629355 5148 server_base.cc:1034] running on GCE node
I20260619 06:34:39.629995 5148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:39.631000 5148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:39.632252 5148 hybrid_clock.cc:648] HybridClock initialized: now 1781850879632209 us; error 47 us; skew 500 ppm
I20260619 06:34:39.632648 5148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:39 dist-test-slave-nf4n krb5kdc[4906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850879, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:39.636134 5148 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:34:39.637759 5148 webserver.cc:460] Webserver started at http://127.28.104.194:35649/ using document root <none> and password file <none>
I20260619 06:34:39.638500 5148 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:39.638655 5148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:39.639003 5148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:39.642216 5148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "24d59aeccb48413591ca5dc5797a104d"
format_stamp: "Formatted at 2026-06-19 06:34:39 on dist-test-slave-nf4n"
server_key: "faa5e4a1435ad52b15dcccdc150263c9"
server_key_iv: "cc7fd6cf10ae516537683a9eeb639679"
server_key_version: "encryptionkey@0"
I20260619 06:34:39.643199 5148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "24d59aeccb48413591ca5dc5797a104d"
format_stamp: "Formatted at 2026-06-19 06:34:39 on dist-test-slave-nf4n"
server_key: "faa5e4a1435ad52b15dcccdc150263c9"
server_key_iv: "cc7fd6cf10ae516537683a9eeb639679"
server_key_version: "encryptionkey@0"
I20260619 06:34:39.648872 5148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:34:39.652683 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:39.654086 5148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:39.654260 5148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "24d59aeccb48413591ca5dc5797a104d"
format_stamp: "Formatted at 2026-06-19 06:34:39 on dist-test-slave-nf4n"
server_key: "faa5e4a1435ad52b15dcccdc150263c9"
server_key_iv: "cc7fd6cf10ae516537683a9eeb639679"
server_key_version: "encryptionkey@0"
I20260619 06:34:39.654452 5148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:39.676290 5148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:39.680707 5148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:39.680997 5148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:39.682053 5148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:39.683385 5148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:39.683462 5148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:39.683597 5148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:39.683663 5148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:39.716022 5148 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:39975
I20260619 06:34:39.716114 5276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:39975 every 8 connection(s)
I20260619 06:34:39.717378 5148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:34:39.717880 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 5148
I20260619 06:34:39.718102 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:34:39.718750 29091 external_mini_cluster.cc:1428] Setting key d08fce8b6970ff013ff6e6f63f2849e3
Jun 19 06:34:39 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850879, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:39.739658 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:39.720600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:46647 (local address 127.28.104.254:37635)
0619 06:34:39.720811 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:39.720818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:39.722273 (+ 1455us) server_negotiation.cc:407] Connection header received
0619 06:34:39.724229 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:39.724238 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:39.724341 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:39.724467 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:39.726044 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:39.727232 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:39.728758 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:39.729669 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:39.729829 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:39.733613 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:39.733618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:39.733621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:39.733653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:39.736246 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:39.737313 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:39.737317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:39.737320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:39.737443 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:39.737897 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:39.737906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:39.737908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:39.738176 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:39.738335 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:39.739074 (+ 739us) server_negotiation.cc:299] Negotiation successful
0619 06:34:39.739304 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:34:39.740664 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:39.721257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:46647)
0619 06:34:39.722102 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:39.722147 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:39.723942 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:39.724673 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:39.724681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:39.725274 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:39.725843 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:39.725860 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:39.727380 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:39.727383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:39.728596 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:39.728607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:39.729878 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:39.729883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:39.730070 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:39.730757 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:39.730779 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:39.733383 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:39.736403 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:39.736416 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:39.736437 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:39.737184 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:39.737570 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:39.737576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:39.737580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:39.737774 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:39.738309 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:39.738318 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:39.738697 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 06:34:39.738969 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 06:34:39.739521 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":338,"threads_started":1}
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
I20260619 06:34:39.742388 5277 heartbeater.cc:347] Connected to a master server at 127.28.104.254:37635
I20260619 06:34:39.742679 5277 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:39.743449 5277 heartbeater.cc:511] Master 127.28.104.254:37635 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
I20260619 06:34:39.745174 4952 ts_manager.cc:194] Registered new tserver with Master: 24d59aeccb48413591ca5dc5797a104d (127.28.104.194:39975)
I20260619 06:34:39.746412 4952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:46647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:34:39.770437 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37635
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:40.106194 5282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:40.106514 5282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:40.106701 5282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:40.114528 5282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:40.114675 5282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:40.114878 5282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:34:40.125425 5282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:40.126820 5282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:40.128286 5282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:40.142545 5287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:40.142513 5288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:40.142643 5290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:40.146523 5282 server_base.cc:1034] running on GCE node
I20260619 06:34:40.147205 5282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:40.148238 5282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:40.149470 5282 hybrid_clock.cc:648] HybridClock initialized: now 1781850880149423 us; error 39 us; skew 500 ppm
I20260619 06:34:40.149833 5282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:40 dist-test-slave-nf4n krb5kdc[4906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850880, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:40.153139 5282 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:34:40.154677 5282 webserver.cc:460] Webserver started at http://127.28.104.195:38769/ using document root <none> and password file <none>
I20260619 06:34:40.155447 5282 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:40.155604 5282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:40.155942 5282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:40.159322 5282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "60b17a17346747ca8521346118c3f131"
format_stamp: "Formatted at 2026-06-19 06:34:40 on dist-test-slave-nf4n"
server_key: "1d54d461b069ab834cda9f5af1a1dfe7"
server_key_iv: "aa775b8648fa05ca0d2a53a6248bfa25"
server_key_version: "encryptionkey@0"
I20260619 06:34:40.160212 5282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "60b17a17346747ca8521346118c3f131"
format_stamp: "Formatted at 2026-06-19 06:34:40 on dist-test-slave-nf4n"
server_key: "1d54d461b069ab834cda9f5af1a1dfe7"
server_key_iv: "aa775b8648fa05ca0d2a53a6248bfa25"
server_key_version: "encryptionkey@0"
I20260619 06:34:40.165743 5282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:34:40.169421 5297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:40.170796 5282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:34:40.170986 5282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "60b17a17346747ca8521346118c3f131"
format_stamp: "Formatted at 2026-06-19 06:34:40 on dist-test-slave-nf4n"
server_key: "1d54d461b069ab834cda9f5af1a1dfe7"
server_key_iv: "aa775b8648fa05ca0d2a53a6248bfa25"
server_key_version: "encryptionkey@0"
I20260619 06:34:40.171204 5282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:40.183538 5282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:40.188010 5282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:40.188324 5282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:40.189353 5282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:40.190671 5282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:40.190745 5282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:40.190912 5282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:40.190960 5282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:40.224927 5282 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:46575
I20260619 06:34:40.224967 5409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:46575 every 8 connection(s)
I20260619 06:34:40.226346 5282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:34:40.231782 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 5282
I20260619 06:34:40.231978 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:34:40.232555 29091 external_mini_cluster.cc:1428] Setting key 377efe4b9a4381a966f0b570db8bf5cd
I20260619 06:34:40.238325 5142 heartbeater.cc:503] Master 127.28.104.254:37635 was elected leader, sending a full tablet report...
Jun 19 06:34:40 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850880, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:34:40.248078 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:40.228942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:46333 (local address 127.28.104.254:37635)
0619 06:34:40.229192 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:40.229198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:40.230310 (+ 1112us) server_negotiation.cc:407] Connection header received
0619 06:34:40.232490 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:40.232495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:40.232565 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:40.232689 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:40.234725 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:40.235895 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:40.237633 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:40.238016 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:40.238125 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:40.242176 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:40.242181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:40.242185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:40.242216 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:40.244991 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:40.245723 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:40.245727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:40.245730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:40.245851 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:40.246238 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:40.246243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:40.246246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:40.246547 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:40.246662 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:40.247490 (+ 828us) server_negotiation.cc:299] Negotiation successful
0619 06:34:40.247727 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 06:34:40.249081 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:40.229491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:46333)
0619 06:34:40.230154 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:40.230191 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:40.232163 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:40.232979 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:40.232990 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:40.233704 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:40.234534 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:40.234560 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:40.236043 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:40.236048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:40.237352 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:40.237364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:40.238251 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:40.238257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:40.238549 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:40.239324 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:40.239347 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:40.241978 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:40.245168 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:40.245175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:40.245188 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:40.245613 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:40.245940 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:34:40.245943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:34:40.245946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:34:40.246116 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:34:40.246675 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:40.246685 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:40.247101 (+ 416us) client_negotiation.cc:769] Sending connection context
0619 06:34:40.247359 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 06:34:40.247849 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":259,"threads_started":1}
I20260619 06:34:40.250380 5410 heartbeater.cc:347] Connected to a master server at 127.28.104.254:37635
I20260619 06:34:40.250703 5410 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:40.251396 5410 heartbeater.cc:511] Master 127.28.104.254:37635 requested a full tablet report, sending...
I20260619 06:34:40.252780 4952 ts_manager.cc:194] Registered new tserver with Master: 60b17a17346747ca8521346118c3f131 (127.28.104.195:46575)
I20260619 06:34:40.253674 4952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:46333
I20260619 06:34:40.260259 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab.
Entry for principal oryx/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab.
WARNING: no policy specified for oryx/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab.
Entry for principal oryx/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab.
WARNING: no policy specified for oryx/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab.
Entry for principal oryx/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab.
I20260619 06:34:40.396006 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5013
Jun 19 06:34:40 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:34:40.412989 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:40.397328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.28.104.193:37651)
0619 06:34:40.397998 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:40.398005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:40.398040 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:34:40.398099 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:40.398105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:40.398323 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:40.398470 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:40.399828 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:40.400959 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:40.402288 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:40.402933 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:40.403045 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:40.407422 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:40.407434 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:40.407447 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:40.407483 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:40.410375 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:40.411078 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:40.411085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:40.411088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:40.411214 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:40.411545 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:40.411548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:40.411550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:40.411875 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:40.412124 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:40.412423 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 06:34:40.412707 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":361,"threads_started":1}
I20260619 06:34:40.414427 5137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:40.497740 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 5013
I20260619 06:34:40.515187 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5148
Jun 19 06:34:40 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:34:40.532127 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:40.516169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.28.104.194:39975)
0619 06:34:40.516810 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:40.516817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:40.516848 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:34:40.516923 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:40.516930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:40.517228 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:40.517376 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:40.518586 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:40.519825 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:40.521458 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:40.522174 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:40.522295 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:40.526378 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:40.526391 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:40.526405 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:40.526451 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:40.529405 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:40.530079 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:40.530087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:40.530089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:40.530221 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:40.530551 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:40.530554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:40.530556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:40.530891 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:40.531129 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:40.531518 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 06:34:40.531800 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":327,"threads_started":1}
I20260619 06:34:40.533430 5272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:40.615923 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 5148
I20260619 06:34:40.632306 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5282
Jun 19 06:34:40 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:34:40.649607 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:40.633630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.28.104.195:46575)
0619 06:34:40.634219 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:40.634226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:40.634254 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:34:40.634352 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:40.634358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:40.634600 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:40.634726 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:40.636083 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:40.637347 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:40.638666 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:40.639268 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:40.639394 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:40.643933 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:40.643943 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:40.643958 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:40.643991 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:40.647005 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:40.647637 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:40.647643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:40.647645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:40.647765 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:40.648101 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:40.648104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:40.648106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:40.648441 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:40.648683 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:40.648918 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:34:40.649167 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":300,"threads_started":1}
I20260619 06:34:40.651055 5405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:40.735472 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 5282
I20260619 06:34:40.751343 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 4919
I20260619 06:34:40.752562 4982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:40.859694 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 4919
I20260619 06:34:40.880276 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:37635
--webserver_interface=127.28.104.254
--webserver_port=46815
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:37635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:34:41.208272 5428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:41.208588 5428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:41.208750 5428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:41.215977 5428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:41.216091 5428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:41.216184 5428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:41.216274 5428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:41.216362 5428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:41.226434 5428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:37635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:37635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=46815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:41.227857 5428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:41.229234 5428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:41.240865 5433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:41.240937 5436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:41.240957 5434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:41.242153 5428 server_base.cc:1034] running on GCE node
I20260619 06:34:41.243065 5428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:41.244954 5428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:41.246251 5428 hybrid_clock.cc:648] HybridClock initialized: now 1781850881246181 us; error 61 us; skew 500 ppm
I20260619 06:34:41.246657 5428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:41 dist-test-slave-nf4n krb5kdc[4906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850881, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:41.249835 5428 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:34:41.251405 5428 webserver.cc:460] Webserver started at http://127.28.104.254:46815/ using document root <none> and password file <none>
I20260619 06:34:41.252151 5428 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:41.252313 5428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:41.258008 5428 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:34:41.260996 5443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:41.262513 5428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:34:41.262740 5428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "29b8aa98065c410f8a1534013a35b550"
format_stamp: "Formatted at 2026-06-19 06:34:38 on dist-test-slave-nf4n"
server_key: "15402af5bd2046cbdd6c65bc6a1a9c55"
server_key_iv: "ebbc989c57dd3fce8a98acfcdf268f52"
server_key_version: "encryptionkey@0"
I20260619 06:34:41.264111 5428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:41.303536 5428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:41.307930 5428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:41.308221 5428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:41.327687 5428 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:37635
I20260619 06:34:41.327711 5495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:37635 every 8 connection(s)
I20260619 06:34:41.329324 5428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:41.331862 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 5428
I20260619 06:34:41.332647 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:37651
--local_ip_for_outbound_sockets=127.28.104.193
--tserver_master_addrs=127.28.104.254:37635
--webserver_port=34475
--webserver_interface=127.28.104.193
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.104.193
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:34:41.336858 5496 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:34:41.341286 5496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550: Bootstrap starting.
I20260619 06:34:41.369693 5496 log.cc:824] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:41.381229 5496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:34:41.381853 5496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550: Bootstrap complete.
I20260619 06:34:41.395634 5496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } }
I20260619 06:34:41.396813 5496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b8aa98065c410f8a1534013a35b550, State: Initialized, Role: FOLLOWER
I20260619 06:34:41.397701 5496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } }
I20260619 06:34:41.398042 5496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:41.398200 5496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:41.398427 5496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:34:41.404770 5496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } }
I20260619 06:34:41.405397 5496 leader_election.cc:302] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b8aa98065c410f8a1534013a35b550; no voters:
I20260619 06:34:41.406052 5496 leader_election.cc:288] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:34:41.406222 5500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:34:41.407430 5500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [term 2 LEADER]: Becoming Leader. State: Replica: 29b8aa98065c410f8a1534013a35b550, State: Running, Role: LEADER
I20260619 06:34:41.408047 5500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } }
I20260619 06:34:41.408174 5496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:41.410924 5501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "29b8aa98065c410f8a1534013a35b550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } } }
I20260619 06:34:41.410923 5502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b8aa98065c410f8a1534013a35b550. Latest consensus state: current_term: 2 leader_uuid: "29b8aa98065c410f8a1534013a35b550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b8aa98065c410f8a1534013a35b550" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37635 } } }
I20260619 06:34:41.411332 5501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:41.411414 5502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:41.422484 5508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:41.428732 5508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:41.430191 5508 catalog_manager.cc:1259] Loaded cluster ID: 3f1c81ea5faf47c1884eb1f6cee70686
I20260619 06:34:41.430353 5508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:41.434315 5508 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:41.436152 5508 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29b8aa98065c410f8a1534013a35b550: Loaded TSK: 0
I20260619 06:34:41.437901 5508 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:34:41.702270 5498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:41.702569 5498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:41.702714 5498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:41.710422 5498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:41.710536 5498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:41.710713 5498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:34:41.720768 5498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.193.keytab
--principal=oryx/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:37651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=34475
--tserver_master_addrs=127.28.104.254:37635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:41.722162 5498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:41.723544 5498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:41.738372 5523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:41.738885 5522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:41.739557 5525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:41.740262 5498 server_base.cc:1034] running on GCE node
I20260619 06:34:41.740971 5498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:41.742117 5498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:41.743371 5498 hybrid_clock.cc:648] HybridClock initialized: now 1781850881743299 us; error 59 us; skew 500 ppm
I20260619 06:34:41.743727 5498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:41 dist-test-slave-nf4n krb5kdc[4906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850881, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:41.751168 5498 init.cc:377] Logged in from keytab as oryx/127.28.104.193@KRBTEST.COM (short username oryx)
I20260619 06:34:41.752858 5498 webserver.cc:460] Webserver started at http://127.28.104.193:34475/ using document root <none> and password file <none>
I20260619 06:34:41.753666 5498 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:41.753782 5498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:41.759709 5498 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 06:34:41.763070 5533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:41.764693 5498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:34:41.764896 5498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "f65b2ee1ae6a48cca42758cb7854ae92"
format_stamp: "Formatted at 2026-06-19 06:34:39 on dist-test-slave-nf4n"
server_key: "bd4d6617930bb46cae0c792490413614"
server_key_iv: "0be4e2345e2b75ab07746308411f229e"
server_key_version: "encryptionkey@0"
I20260619 06:34:41.765997 5498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:41.779281 5498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:41.783681 5498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:41.783967 5498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:41.784981 5498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:41.786497 5498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:41.786624 5498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:41.786742 5498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:41.786827 5498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:41.822180 5498 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:37651
I20260619 06:34:41.822268 5646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:37651 every 8 connection(s)
I20260619 06:34:41.823776 5498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:34:41.829954 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 5498
I20260619 06:34:41.830696 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:39975
--local_ip_for_outbound_sockets=127.28.104.194
--tserver_master_addrs=127.28.104.254:37635
--webserver_port=35649
--webserver_interface=127.28.104.194
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.104.194
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:34:41 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:41 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:41.843988 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:41.826954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:47587)
0619 06:34:41.827793 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:41.827838 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:41.829632 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:41.831308 (+ 1676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:41.831316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:41.831824 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:41.832371 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:41.832388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.834779 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.834787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:41.836696 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:41.836713 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.837976 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.837984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:41.838325 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:41.839113 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:41.839138 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:41.843010 (+ 3872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":337,"threads_started":1}
W20260619 06:34:41.844708 5647 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:37635 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.104.254:37635: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
W20260619 06:34:41.844818 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:41.826855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:47587 (local address 127.28.104.254:37635)
0619 06:34:41.827631 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:41.827642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:41.827972 (+ 330us) server_negotiation.cc:407] Connection header received
0619 06:34:41.830269 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:41.830289 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:41.830768 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:41.831203 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:41.832586 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.834585 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.836852 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.837727 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.837865 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:41.843531 (+ 5666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:47587: BlockingRecv error: recv got EOF from 127.28.104.193:47587 (error 108)
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":313,"threads_started":1}
Jun 19 06:34:41 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:41 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:41.854210 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:41.845314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:48713)
0619 06:34:41.845568 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:41.845583 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:41.845665 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:41.846044 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:41.846047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:41.846332 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:41.846498 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:41.846505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.848104 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.848108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:41.848968 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:41.848978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.850216 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.850220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:41.850347 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:41.850949 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:41.850967 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:41.854009 (+ 3042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129}
W20260619 06:34:41.854688 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:41.845418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:48713 (local address 127.28.104.254:37635)
0619 06:34:41.845639 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:41.845648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:41.845673 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:34:41.845774 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:41.845782 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:41.845892 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:41.846015 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:41.846654 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.847959 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.849224 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.850058 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.850230 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:41.854334 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:48713: BlockingRecv error: recv got EOF from 127.28.104.193:48713 (error 108)
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":86}
Jun 19 06:34:41 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:41 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:41.862728 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:41.854999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:50749)
0619 06:34:41.855168 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:41.855187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:41.855286 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:41.855603 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:41.855608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:41.855913 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:41.856057 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:41.856065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.857125 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.857132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:41.858039 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:41.858050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.858763 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.858769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:41.858938 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:41.859542 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:41.859559 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:41.862423 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 06:34:41.862926 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:41.855133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:50749 (local address 127.28.104.254:37635)
0619 06:34:41.855287 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:41.855293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:41.855310 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:34:41.855390 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:41.855395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:41.855474 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:41.855582 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:41.856191 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.856945 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.858187 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:41.858614 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:41.858741 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:41.862651 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:50749: BlockingRecv error: recv got EOF from 127.28.104.193:50749 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 06:34:41.863070 5647 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:34:42.175976 5651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:42.176314 5651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:42.176481 5651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:42.183738 5651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:42.183913 5651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:42.184090 5651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:34:42.194041 5651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.194.keytab
--principal=oryx/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:39975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=35649
--tserver_master_addrs=127.28.104.254:37635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:42.195574 5651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:42.197000 5651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:42.210822 5657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:42.210803 5656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:42.210880 5659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:42.211563 5651 server_base.cc:1034] running on GCE node
I20260619 06:34:42.212188 5651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:42.213184 5651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:42.214509 5651 hybrid_clock.cc:648] HybridClock initialized: now 1781850882214455 us; error 44 us; skew 500 ppm
I20260619 06:34:42.214867 5651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850882, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:42.218509 5651 init.cc:377] Logged in from keytab as oryx/127.28.104.194@KRBTEST.COM (short username oryx)
I20260619 06:34:42.220131 5651 webserver.cc:460] Webserver started at http://127.28.104.194:35649/ using document root <none> and password file <none>
I20260619 06:34:42.220894 5651 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:42.221076 5651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:42.227174 5651 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 06:34:42.230070 5666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:42.231511 5651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:34:42.231703 5651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "24d59aeccb48413591ca5dc5797a104d"
format_stamp: "Formatted at 2026-06-19 06:34:39 on dist-test-slave-nf4n"
server_key: "faa5e4a1435ad52b15dcccdc150263c9"
server_key_iv: "cc7fd6cf10ae516537683a9eeb639679"
server_key_version: "encryptionkey@0"
I20260619 06:34:42.232753 5651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:42.250595 5651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:42.254792 5651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:42.255072 5651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:42.256084 5651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:42.257463 5651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:42.257587 5651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:42.257704 5651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:42.257790 5651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:42.291328 5651 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:39975
I20260619 06:34:42.291383 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:39975 every 8 connection(s)
I20260619 06:34:42.292891 5651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:34:42.297223 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 5651
I20260619 06:34:42.297832 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:46575
--local_ip_for_outbound_sockets=127.28.104.195
--tserver_master_addrs=127.28.104.254:37635
--webserver_port=38769
--webserver_interface=127.28.104.195
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.104.195
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:34:42.309612 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:42.295566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:57695 (local address 127.28.104.254:37635)
0619 06:34:42.295747 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:42.295752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:42.296878 (+ 1126us) server_negotiation.cc:407] Connection header received
0619 06:34:42.298915 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:42.298921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:42.299021 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:42.299139 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:42.300935 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.301754 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.303400 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.303835 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.303976 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.309303 (+ 5327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:57695: BlockingRecv error: recv got EOF from 127.28.104.194:57695 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:34:42.310349 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:42.296028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:57695)
0619 06:34:42.296715 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:42.296756 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:42.298625 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:42.299437 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:42.299446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:42.300031 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:42.300734 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.300755 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.301941 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.301946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.303238 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.303250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.304086 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.304094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.304342 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.305088 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:42.305114 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:42.309085 (+ 3971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":268,"threads_started":1}
W20260619 06:34:42.311003 5780 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:37635 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.104.254:37635: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:42.319651 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:42.311629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:35857)
0619 06:34:42.311799 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:42.311819 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:42.311931 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:42.312302 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:42.312306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:42.312732 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:42.312899 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.312907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.313908 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.313914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.314806 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.314816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.315505 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.315511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.315684 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.316298 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:42.316316 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:42.319352 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 06:34:42.319907 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:42.311723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:35857 (local address 127.28.104.254:37635)
0619 06:34:42.311904 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:42.311909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:42.311927 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:34:42.312099 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:42.312104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:42.312177 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:42.312281 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:42.313064 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.313772 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.314942 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.315357 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.315492 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.319559 (+ 4067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:35857: BlockingRecv error: recv got EOF from 127.28.104.194:35857 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:42.329487 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:42.320482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:56573)
0619 06:34:42.320662 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:42.320682 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:42.320769 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:42.321128 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:42.321131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:42.321384 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:42.321519 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.321528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.322925 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.322931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.323858 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.323870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.324807 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.324812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.324990 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.325798 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:42.325842 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:42.329177 (+ 3335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:34:42.329648 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:42.320607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:56573 (local address 127.28.104.254:37635)
0619 06:34:42.320807 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:42.320814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:42.320833 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:34:42.320884 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:42.320889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:42.320970 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:42.321110 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:42.321647 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.322739 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.323996 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.324630 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.324750 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.329421 (+ 4671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:56573: BlockingRecv error: recv got EOF from 127.28.104.194:56573 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 06:34:42.330041 5780 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:34:42.632025 5783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:42.632357 5783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:42.632551 5783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:42.640210 5783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:42.640368 5783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:42.640544 5783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:34:42.650477 5783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/krb5kdc/oryx_127.28.104.195.keytab
--principal=oryx/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:46575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=38769
--tserver_master_addrs=127.28.104.254:37635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:42.651867 5783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:42.653326 5783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:42.666240 5788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:42.666198 5789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:42.666513 5791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:42.667460 5783 server_base.cc:1034] running on GCE node
I20260619 06:34:42.668076 5783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:42.669090 5783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:42.670315 5783 hybrid_clock.cc:648] HybridClock initialized: now 1781850882670268 us; error 39 us; skew 500 ppm
I20260619 06:34:42.670646 5783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850882, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:34:42.673744 5783 init.cc:377] Logged in from keytab as oryx/127.28.104.195@KRBTEST.COM (short username oryx)
I20260619 06:34:42.675261 5783 webserver.cc:460] Webserver started at http://127.28.104.195:38769/ using document root <none> and password file <none>
I20260619 06:34:42.675959 5783 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:42.676115 5783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:42.681500 5783 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:42.684195 5798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:42.685580 5783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:42.685751 5783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "60b17a17346747ca8521346118c3f131"
format_stamp: "Formatted at 2026-06-19 06:34:40 on dist-test-slave-nf4n"
server_key: "1d54d461b069ab834cda9f5af1a1dfe7"
server_key_iv: "aa775b8648fa05ca0d2a53a6248bfa25"
server_key_version: "encryptionkey@0"
I20260619 06:34:42.686709 5783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:42.712459 5783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:42.717105 5783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:42.717384 5783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:42.718394 5783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:42.719727 5783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:42.719831 5783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:42.719985 5783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:42.720072 5783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:42.752579 5783 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:46575
I20260619 06:34:42.752671 5910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:46575 every 8 connection(s)
I20260619 06:34:42.754119 5783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:34:42.761500 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 5783
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:34:42.772619 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:42.756812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:34561 (local address 127.28.104.254:37635)
0619 06:34:42.756981 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:42.756985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:42.758267 (+ 1282us) server_negotiation.cc:407] Connection header received
0619 06:34:42.760211 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:42.760216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:42.760310 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:42.760439 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:42.762530 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.763729 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.766122 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.766593 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.766738 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.772330 (+ 5592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:34561: BlockingRecv error: recv got EOF from 127.28.104.195:34561 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:34:42.773432 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:42.757329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:34561)
0619 06:34:42.758048 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:42.758091 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:42.759934 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:42.760680 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:42.760689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:42.761417 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:42.762244 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.762264 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.763872 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.763880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.765978 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.765990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.766823 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.766830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.767063 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.767827 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:42.767850 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:42.772158 (+ 4308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":285,"threads_started":1}
W20260619 06:34:42.774091 5911 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:37635 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.104.254:37635: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:42.782382 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:42.762626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.28.104.254:37635)
0619 06:34:42.763461 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:42.763468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:42.763488 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:34:42.763542 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:42.763547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:42.763617 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:42.763749 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:42.765799 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.766599 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.767919 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.768382 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.768549 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.771599 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:42.771607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:42.771625 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:42.771659 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:42.776536 (+ 4877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:42.777900 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:42.777913 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:42.777923 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:42.778182 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:34:42.779357 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:34:42.779361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:34:42.779364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:34:42.780033 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:42.780340 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:42.781699 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0619 06:34:42.782001 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":599,"threads_started":1}
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:42.785072 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:42.774679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:46383)
0619 06:34:42.774932 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:42.774951 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:42.775064 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:42.775406 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:42.775413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:42.775736 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:42.775865 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.775873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.776820 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.776825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.778299 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.778313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.779205 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.779211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.779395 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.780012 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:42.780033 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:42.784752 (+ 4719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132}
W20260619 06:34:42.785190 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:42.774823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:46383 (local address 127.28.104.254:37635)
0619 06:34:42.775005 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:42.775012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:42.775032 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:34:42.775203 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:42.775208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:42.775262 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:42.775374 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:42.776001 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.776687 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.778470 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.779057 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.779193 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.784936 (+ 5743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:46383: BlockingRecv error: recv got EOF from 127.28.104.195:46383 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:42.793883 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:42.786229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:59511)
0619 06:34:42.786409 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:42.786429 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:42.786529 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:42.787011 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:42.787015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:42.787292 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:42.787430 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.787438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.788352 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.788356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.789425 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.789437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.790134 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.790139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.790257 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.790812 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:42.790828 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:42.793585 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 06:34:42.794031 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:42.786286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:59511 (local address 127.28.104.254:37635)
0619 06:34:42.786492 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:42.786496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:42.786511 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:34:42.786757 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:42.786763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:42.786848 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:42.786930 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:42.787547 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.788243 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.789542 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.789995 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.790085 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.793773 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:59511: BlockingRecv error: recv got EOF from 127.28.104.195:59511 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260619 06:34:42.794380 5911 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:42 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:42.873311 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:42.863831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:55817)
0619 06:34:42.864376 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:42.864396 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:42.864531 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:42.864876 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:42.864893 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:42.865432 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:42.865644 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.865654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.866731 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.866735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.867846 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:42.867860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.868673 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.868678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:42.868810 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.869554 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:42.869575 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:42.873001 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":294,"threads_started":1}
W20260619 06:34:42.873428 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:42.863875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:55817 (local address 127.28.104.254:37635)
0619 06:34:42.864043 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:42.864048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:42.864486 (+ 438us) server_negotiation.cc:407] Connection header received
0619 06:34:42.864656 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:42.864660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:42.864722 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:42.864806 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:42.865796 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.866587 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.868018 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:42.868494 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:42.868603 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:42.873256 (+ 4653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:55817: BlockingRecv error: recv got EOF from 127.28.104.193:55817 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:34:43 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:43 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:43.340543 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:43.330731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:58267)
0619 06:34:43.331636 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:43.331656 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:43.331878 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:43.332343 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:43.332348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:43.332932 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:43.333246 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:43.333260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.334735 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.334741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:43.335830 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:43.335843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.336630 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.336637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:43.336833 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:43.337443 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:43.337460 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:43.340313 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":504,"threads_started":1}
W20260619 06:34:43.340734 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:43.330798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:58267 (local address 127.28.104.254:37635)
0619 06:34:43.331061 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:43.331071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:43.331768 (+ 697us) server_negotiation.cc:407] Connection header received
0619 06:34:43.332075 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:43.332083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:43.332165 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:43.332268 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:43.333402 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.334581 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.335977 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.336427 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.336597 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:43.340508 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:58267: BlockingRecv error: recv got EOF from 127.28.104.194:58267 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 06:34:43 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:43 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:43.803371 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:43.795098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:42087)
0619 06:34:43.795780 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:43.795800 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:43.795965 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:43.796372 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:43.796377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:43.796925 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:43.797195 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:43.797205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.798161 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.798165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:43.799117 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:43.799128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.799784 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.799789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:43.799907 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:43.800484 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:43.800501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:43.803151 (+ 2650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":471,"threads_started":1}
W20260619 06:34:43.803556 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:43.795209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:42087 (local address 127.28.104.254:37635)
0619 06:34:43.795432 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:43.795440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:43.795912 (+ 472us) server_negotiation.cc:407] Connection header received
0619 06:34:43.796108 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:43.796113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:43.796194 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:43.796286 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:43.797326 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.798038 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.799235 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.799640 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.799729 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:43.803340 (+ 3611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:42087: BlockingRecv error: recv got EOF from 127.28.104.195:42087 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 06:34:43 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:43 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:43.883035 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:43.874283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:41355)
0619 06:34:43.874919 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:43.874934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:43.875085 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:43.875427 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:43.875432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:43.875873 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:43.876043 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:43.876052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.877114 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.877121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:43.878016 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:43.878027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.878745 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.878752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:43.878923 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:43.879567 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:43.879585 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:43.882682 (+ 3097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":282,"threads_started":1}
W20260619 06:34:43.883132 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:43.874539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:41355 (local address 127.28.104.254:37635)
0619 06:34:43.874845 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:43.874851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:43.875014 (+ 163us) server_negotiation.cc:407] Connection header received
0619 06:34:43.875217 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:43.875222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:43.875295 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:43.875398 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:43.876190 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.876943 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.878160 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:43.878588 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:43.878727 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:43.882870 (+ 4143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:41355: BlockingRecv error: recv got EOF from 127.28.104.193:41355 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
Jun 19 06:34:44 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:44 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:44.350811 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:44.341591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:56375)
0619 06:34:44.342213 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:44.342232 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:44.342405 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:44.342754 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:44.342758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:44.343198 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:44.343401 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:44.343411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.344725 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.344731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:44.346151 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:44.346168 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.346862 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.346867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:44.346980 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:44.347552 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:44.347570 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:44.350563 (+ 2993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":348,"threads_started":1}
W20260619 06:34:44.351011 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:44.341744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:56375 (local address 127.28.104.254:37635)
0619 06:34:44.341995 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:44.342004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:44.342329 (+ 325us) server_negotiation.cc:407] Connection header received
0619 06:34:44.342538 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:44.342543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:44.342612 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:44.342719 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:44.343571 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.344473 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.346297 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.346721 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.346821 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:44.350781 (+ 3960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:56375: BlockingRecv error: recv got EOF from 127.28.104.194:56375 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 19 06:34:44 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:44 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:44.812857 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:44.804404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:52451)
0619 06:34:44.804949 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:44.804970 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:44.805110 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:44.805398 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:44.805401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:44.805837 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:44.806006 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:44.806014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.807032 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.807037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:44.808140 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:44.808154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.809058 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.809068 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:44.809199 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:44.809858 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:44.809874 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:44.812511 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":309,"threads_started":1}
W20260619 06:34:44.812943 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:44.804692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:52451 (local address 127.28.104.254:37635)
0619 06:34:44.804867 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:44.804871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:44.805042 (+ 171us) server_negotiation.cc:407] Connection header received
0619 06:34:44.805206 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:44.805210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:44.805270 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:44.805356 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:44.806124 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.806864 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.808284 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.808783 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.808946 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:44.812696 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:52451: BlockingRecv error: recv got EOF from 127.28.104.195:52451 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 06:34:44 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:44 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:44.892453 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:44.884008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:44459)
0619 06:34:44.884526 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:44.884542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:44.884655 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:44.884948 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:44.884953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:44.885354 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:44.885560 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:44.885569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.886565 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.886568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:44.887454 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:44.887465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.888090 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.888093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:44.888208 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:44.888780 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:44.888797 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:44.892089 (+ 3292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":290,"threads_started":1}
W20260619 06:34:44.892536 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:44.884059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:44459 (local address 127.28.104.254:37635)
0619 06:34:44.884247 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:44.884252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:44.884616 (+ 364us) server_negotiation.cc:407] Connection header received
0619 06:34:44.884751 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:44.884754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:44.884813 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:44.884896 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:44.885695 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.886447 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.887583 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:44.887972 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:44.888072 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:44.892291 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:44459: BlockingRecv error: recv got EOF from 127.28.104.193:44459 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 06:34:45 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:45 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:45.360673 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:45.351855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:35201)
0619 06:34:45.352444 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:45.352466 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:45.352580 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:45.352881 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:45.352885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:45.353342 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:45.353508 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:45.353516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.354706 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.354711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:45.355898 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:45.355912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.356610 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.356618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:45.356790 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:45.357496 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:45.357526 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:45.360408 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":360,"threads_started":1}
W20260619 06:34:45.361093 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:45.351965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:35201 (local address 127.28.104.254:37635)
0619 06:34:45.352149 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:45.352153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:45.352553 (+ 400us) server_negotiation.cc:407] Connection header received
0619 06:34:45.352685 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:45.352688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:45.352750 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:45.352858 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:45.353656 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.354548 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.356052 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.356451 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.356629 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:45.360758 (+ 4129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:35201: BlockingRecv error: recv got EOF from 127.28.104.194:35201 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 06:34:45 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:45 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:45.822305 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:45.813937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:46631)
0619 06:34:45.814482 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:45.814514 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:45.814679 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:45.815047 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:45.815052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:45.815511 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:45.815679 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:45.815688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.816671 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.816679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:45.817604 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:45.817615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.818298 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.818303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:45.818476 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:45.819054 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:45.819072 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:45.821952 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":295,"threads_started":1}
W20260619 06:34:45.822382 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:45.814122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:46631 (local address 127.28.104.254:37635)
0619 06:34:45.814355 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:45.814379 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:45.814607 (+ 228us) server_negotiation.cc:407] Connection header received
0619 06:34:45.814811 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:45.814818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:45.814888 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:45.814991 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:45.815828 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.816546 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.817749 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.818150 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.818285 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:45.822153 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:46631: BlockingRecv error: recv got EOF from 127.28.104.195:46631 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 06:34:45 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:45 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:45.901775 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:45.893505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:45891)
0619 06:34:45.893939 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:45.893956 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:45.894082 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:45.894375 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:45.894378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:45.894978 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:45.895144 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:45.895153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.896101 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.896105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:45.897156 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:45.897168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.897806 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.897810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:45.897922 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:45.898499 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:45.898515 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:45.901555 (+ 3040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":242,"threads_started":1}
W20260619 06:34:45.901957 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:45.893553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:45891 (local address 127.28.104.254:37635)
0619 06:34:45.893783 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:45.893790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:45.894032 (+ 242us) server_negotiation.cc:407] Connection header received
0619 06:34:45.894180 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:45.894183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:45.894246 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:45.894327 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:45.895279 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.895993 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.897275 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:45.897690 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:45.897791 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:45.901749 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:45891: BlockingRecv error: recv got EOF from 127.28.104.193:45891 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 06:34:46 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:46 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:46.370704 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:46.361742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:59681)
0619 06:34:46.362505 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:46.362526 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:46.362705 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:46.363158 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:46.363163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:46.363647 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:46.363801 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:46.363811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.364832 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.364836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:46.365894 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:46.365907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.366580 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.366584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:46.366708 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:46.367298 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:46.367316 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:46.370376 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":493,"threads_started":1}
W20260619 06:34:46.370851 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:46.361828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:59681 (local address 127.28.104.254:37635)
0619 06:34:46.362039 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:46.362046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:46.362620 (+ 574us) server_negotiation.cc:407] Connection header received
0619 06:34:46.362877 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:46.362883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:46.362986 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:46.363099 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:46.363937 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.364709 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.366026 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.366453 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.366549 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:46.370626 (+ 4077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:59681: BlockingRecv error: recv got EOF from 127.28.104.194:59681 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 06:34:46 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:46 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:34:46.837633 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:46.823606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:56565 (local address 127.28.104.254:37635)
0619 06:34:46.823803 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:46.823808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:46.824399 (+ 591us) server_negotiation.cc:407] Connection header received
0619 06:34:46.824687 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:46.824693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:46.824781 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:46.824878 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:46.825680 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.826546 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.827788 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.828181 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.828271 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:46.837364 (+ 9093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:56565: BlockingRecv error: recv got EOF from 127.28.104.195:56565 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:34:46.838189 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:46.823506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:56565)
0619 06:34:46.824281 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:46.824305 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:46.824531 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:46.824957 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:46.824962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:46.825383 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:46.825547 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:46.825557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.826689 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.826694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:46.827666 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:46.827678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.828306 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.828310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:46.828446 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:46.833840 (+ 5394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:46.833862 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:46.837894 (+ 4032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":487,"threads_started":1}
Jun 19 06:34:46 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:46 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:46.911470 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:46.902792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:36685)
0619 06:34:46.903309 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:46.903324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:46.903477 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:46.903825 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:46.903830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:46.904268 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:46.904427 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:46.904435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.905515 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.905522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:46.906446 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:46.906457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.907157 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.907163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:46.907340 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:46.908041 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:46.908075 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:46.911224 (+ 3149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":258,"threads_started":1}
W20260619 06:34:46.911640 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:46.902879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:36685 (local address 127.28.104.254:37635)
0619 06:34:46.903127 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:46.903135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:46.903412 (+ 277us) server_negotiation.cc:407] Connection header received
0619 06:34:46.903622 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:46.903627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:46.903694 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:46.903795 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:46.904579 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.905381 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.906596 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:46.907009 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:46.907144 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:46.911395 (+ 4251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:36685: BlockingRecv error: recv got EOF from 127.28.104.193:36685 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 06:34:47 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:47 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:47.380755 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:47.371734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:40497)
0619 06:34:47.372314 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:47.372334 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:47.372500 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:47.372897 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:47.372902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:47.373519 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:47.373746 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:47.373763 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.375139 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.375143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:47.376053 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:47.376091 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.376895 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.376900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:47.377017 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:47.377678 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:47.377695 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:47.380496 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":270,"threads_started":1}
W20260619 06:34:47.381064 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:47.371989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:40497 (local address 127.28.104.254:37635)
0619 06:34:47.372220 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:47.372228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:47.372420 (+ 192us) server_negotiation.cc:407] Connection header received
0619 06:34:47.372645 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:47.372651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:47.372729 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:47.372838 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:47.373953 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.375012 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.376264 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.376771 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.376875 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:47.380699 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:40497: BlockingRecv error: recv got EOF from 127.28.104.194:40497 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 06:34:47 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:47 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:47.847710 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:47.839108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:36091)
0619 06:34:47.839651 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:47.839672 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:47.839783 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:47.840066 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:47.840069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:47.840453 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:47.840609 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:47.840617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.841657 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.841661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:47.842575 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:47.842588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.843233 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.843237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:47.843358 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:47.843954 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:47.843971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:47.847296 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":263,"threads_started":1}
W20260619 06:34:47.847780 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:47.839313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:36091 (local address 127.28.104.254:37635)
0619 06:34:47.839522 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:47.839528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:47.839750 (+ 222us) server_negotiation.cc:407] Connection header received
0619 06:34:47.839879 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:47.839883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:47.839942 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:47.840024 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:47.840734 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.841524 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.842695 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.843103 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.843192 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:47.847507 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:36091: BlockingRecv error: recv got EOF from 127.28.104.195:36091 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 06:34:47 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:47 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:47.920986 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:47.912403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:39893)
0619 06:34:47.912957 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:47.912976 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:47.913121 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:47.913423 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:47.913426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:47.913856 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:47.914013 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:47.914022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.915016 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.915020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:47.916018 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:47.916032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.916708 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.916712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:47.916838 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:47.917499 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:47.917519 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:47.920776 (+ 3257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":319,"threads_started":1}
W20260619 06:34:47.921303 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:47.912434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:39893 (local address 127.28.104.254:37635)
0619 06:34:47.912626 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:47.912630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:47.913051 (+ 421us) server_negotiation.cc:407] Connection header received
0619 06:34:47.913230 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:47.913233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:47.913292 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:47.913376 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:47.914141 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.914872 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.916160 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:47.916556 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:47.916650 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:47.920964 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:39893: BlockingRecv error: recv got EOF from 127.28.104.193:39893 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 06:34:48 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:48 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:48.390273 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:48.381821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:33085)
0619 06:34:48.382412 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:48.382437 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:48.382578 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:48.382939 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:48.382944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:48.383423 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:48.383611 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:48.383620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.384639 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.384646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:48.385594 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:48.385606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.386317 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.386323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:48.386534 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:48.387118 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:48.387136 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:48.390063 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":317,"threads_started":1}
W20260619 06:34:48.390544 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:48.381944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:33085 (local address 127.28.104.254:37635)
0619 06:34:48.382186 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:48.382192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:48.382537 (+ 345us) server_negotiation.cc:407] Connection header received
0619 06:34:48.382699 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:48.382706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:48.382788 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:48.382901 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:48.383763 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.384500 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.385742 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.386159 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.386298 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:48.390276 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:33085: BlockingRecv error: recv got EOF from 127.28.104.194:33085 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 06:34:48 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:48 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:34:48.858855 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:48.848981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:54725 (local address 127.28.104.254:37635)
0619 06:34:48.849232 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:48.849239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:48.849515 (+ 276us) server_negotiation.cc:407] Connection header received
0619 06:34:48.849704 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:48.849709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:48.849769 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:48.849872 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:48.850626 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.851470 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.852854 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.853548 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.853663 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:48.858586 (+ 4923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:54725: BlockingRecv error: recv got EOF from 127.28.104.195:54725 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 06:34:48.859437 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:48.848838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:54725)
0619 06:34:48.849403 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:48.849423 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:48.849575 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:48.849901 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:48.849906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:48.850318 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:48.850474 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:48.850482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.851638 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.851650 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:48.852707 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:48.852720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.853868 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.853874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:48.854044 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:48.854764 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:48.854788 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:48.859162 (+ 4374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":340,"threads_started":1}
Jun 19 06:34:48 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:48 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:48.931785 5935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:48.922075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:53535)
0619 06:34:48.922740 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:48.922770 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:48.922944 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:48.923607 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:48.923613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:48.924217 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:48.924390 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:48.924400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.925503 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.925509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:48.926449 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:48.926461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.927180 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.927186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:48.927360 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:48.928036 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:48.928053 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:48.931413 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":328,"threads_started":1}
W20260619 06:34:48.931864 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:48.922207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:53535 (local address 127.28.104.254:37635)
0619 06:34:48.922460 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:48.922469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:48.922869 (+ 400us) server_negotiation.cc:407] Connection header received
0619 06:34:48.923131 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:48.923137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:48.923245 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:48.923300 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:48.924540 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.925353 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.926604 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:48.927018 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:48.927151 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:48.931617 (+ 4466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:53535: BlockingRecv error: recv got EOF from 127.28.104.193:53535 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 06:34:49 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:49 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:49.399552 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:49.391295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:51439)
0619 06:34:49.391941 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:49.391961 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:49.392084 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:49.392392 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:49.392395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:49.392807 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:49.392974 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:49.392983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.394056 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.394060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:49.395019 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:49.395031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.395693 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.395697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:49.395815 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:49.396386 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:49.396403 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:49.399299 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":310,"threads_started":1}
W20260619 06:34:49.399747 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:49.391423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:51439 (local address 127.28.104.254:37635)
0619 06:34:49.391665 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:49.391670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:49.392029 (+ 359us) server_negotiation.cc:407] Connection header received
0619 06:34:49.392195 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:49.392199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:49.392260 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:49.392334 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:49.393149 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.393928 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.395160 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.395544 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.395654 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:49.399498 (+ 3844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:51439: BlockingRecv error: recv got EOF from 127.28.104.194:51439 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Jun 19 06:34:49 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:49 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:49.869867 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:49.860567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:45489)
0619 06:34:49.861218 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:49.861239 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:49.861402 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:49.861768 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:49.861772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:49.862259 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:49.862428 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:49.862437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.863507 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.863512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:49.864447 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:49.864459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.865265 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.865271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:49.865457 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:49.866071 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:49.866088 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:49.869601 (+ 3513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":309,"threads_started":1}
W20260619 06:34:49.870042 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:49.860689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:45489 (local address 127.28.104.254:37635)
0619 06:34:49.860884 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:49.860888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:49.861332 (+ 444us) server_negotiation.cc:407] Connection header received
0619 06:34:49.861557 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:49.861561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:49.861625 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:49.861736 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:49.862585 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.863352 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.864599 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.865112 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.865248 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:49.869812 (+ 4564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:45489: BlockingRecv error: recv got EOF from 127.28.104.195:45489 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 06:34:49 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:49 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:49.942193 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:49.932996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:60111)
0619 06:34:49.933834 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:49.933853 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:49.933997 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:49.934417 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:49.934424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:49.934966 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:49.935190 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:49.935208 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.936405 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.936409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:49.937478 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:49.937490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.938282 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.938286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:49.938393 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:49.938988 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:49.939006 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:49.941923 (+ 2917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":551,"threads_started":1}
W20260619 06:34:49.942426 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:49.933276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:60111 (local address 127.28.104.254:37635)
0619 06:34:49.933509 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:49.933517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:49.933972 (+ 455us) server_negotiation.cc:407] Connection header received
0619 06:34:49.934141 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:49.934146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:49.934229 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:49.934358 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:49.935378 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.936268 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.937629 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:49.938123 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:49.938216 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:49.942165 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:60111: BlockingRecv error: recv got EOF from 127.28.104.193:60111 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 06:34:50 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:50 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:50.409324 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:50.400531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:47755)
0619 06:34:50.401101 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:50.401121 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:50.401301 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:50.401647 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:50.401653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:50.402058 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:50.402220 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:50.402230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.403406 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.403419 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:50.404333 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:50.404344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.405115 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.405121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:50.405284 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:50.405883 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:50.405901 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:50.408931 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":289,"threads_started":1}
W20260619 06:34:50.409475 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:50.400656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:47755 (local address 127.28.104.254:37635)
0619 06:34:50.400883 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:50.400890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:50.401230 (+ 340us) server_negotiation.cc:407] Connection header received
0619 06:34:50.401434 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:50.401440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:50.401511 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:50.401615 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:50.402366 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.403250 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.404476 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.404904 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.405061 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:50.409203 (+ 4142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:47755: BlockingRecv error: recv got EOF from 127.28.104.194:47755 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 06:34:50 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:50 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:50.879550 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:50.871005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:57165)
0619 06:34:50.871609 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:50.871660 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:50.871820 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:50.872219 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:50.872224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:50.872767 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:50.872948 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:50.872958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.873986 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.873990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:50.875005 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:50.875016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.875734 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.875739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:50.875855 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:50.876461 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:50.876478 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:50.879226 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":354,"threads_started":1}
W20260619 06:34:50.879716 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:50.871116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:57165 (local address 127.28.104.254:37635)
0619 06:34:50.871335 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:50.871346 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:50.871758 (+ 412us) server_negotiation.cc:407] Connection header received
0619 06:34:50.871962 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:50.871967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:50.872044 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:50.872140 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:50.873107 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.873842 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.875149 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.875540 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.875639 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:50.879436 (+ 3797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:57165: BlockingRecv error: recv got EOF from 127.28.104.195:57165 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 06:34:50 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:50 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:50.951694 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:50.943382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:51199)
0619 06:34:50.944005 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:50.944021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:50.944129 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:50.944396 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:50.944400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:50.944803 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:50.944968 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:50.944976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.945976 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.945979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:50.946998 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:50.947009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.947681 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.947685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:50.947809 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:50.948426 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:50.948443 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:50.951448 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":360,"threads_started":1}
W20260619 06:34:50.951851 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:50.943431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:51199 (local address 127.28.104.254:37635)
0619 06:34:50.943647 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:50.943654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:50.944095 (+ 441us) server_negotiation.cc:407] Connection header received
0619 06:34:50.944211 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:50.944214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:50.944273 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:50.944352 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:50.945149 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.945857 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.947123 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:50.947541 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:50.947636 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:50.951635 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:51199: BlockingRecv error: recv got EOF from 127.28.104.193:51199 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 06:34:51 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:51 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:51.420429 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:51.410346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:34399)
0619 06:34:51.410906 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:51.410928 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:51.411076 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:51.411428 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:51.411433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:51.411900 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:51.412123 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:51.412135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.413918 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.413923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:51.415400 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:51.415436 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.416371 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.416376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:51.416516 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:51.417143 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:51.417162 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:51.420070 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":318,"threads_started":1}
W20260619 06:34:51.420514 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:51.410442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:34399 (local address 127.28.104.254:37635)
0619 06:34:51.410733 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:51.410739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:51.411034 (+ 295us) server_negotiation.cc:407] Connection header received
0619 06:34:51.411179 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:51.411183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:51.411246 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:51.411328 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:51.412391 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.413753 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.415555 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.416221 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.416375 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:51.420264 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:34399: BlockingRecv error: recv got EOF from 127.28.104.194:34399 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Jun 19 06:34:51 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:51 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:51.893620 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:51.880594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:35403)
0619 06:34:51.881139 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:51.881161 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:51.881325 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:51.881683 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:51.881687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:51.882102 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:51.882264 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:51.882273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.883359 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.883367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:51.884443 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:51.884456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.885277 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.885281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:51.885411 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:51.890421 (+ 5010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:51.890441 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:51.893377 (+ 2936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":270,"threads_started":1}
W20260619 06:34:51.893806 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:51.880615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:35403 (local address 127.28.104.254:37635)
0619 06:34:51.880843 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:51.880850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:51.881254 (+ 404us) server_negotiation.cc:407] Connection header received
0619 06:34:51.881468 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:51.881472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:51.881533 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:51.881657 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:51.882410 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.883246 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.884589 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.885100 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.885251 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:51.893600 (+ 8349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:35403: BlockingRecv error: recv got EOF from 127.28.104.195:35403 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 06:34:51 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:51 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:51.962322 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:51.952780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:50657)
0619 06:34:51.953511 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:51.953531 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:51.953725 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:51.954146 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:51.954153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:51.954588 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:51.954746 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:51.954756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.955877 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.955887 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:51.957121 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:51.957139 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.958110 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.958137 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:51.958343 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:51.959056 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:51.959076 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:51.962095 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":380,"threads_started":1}
W20260619 06:34:51.962687 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:51.952959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:50657 (local address 127.28.104.254:37635)
0619 06:34:51.953237 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:51.953246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:51.953653 (+ 407us) server_negotiation.cc:407] Connection header received
0619 06:34:51.953869 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:51.953875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:51.953985 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:51.954111 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:51.954920 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.955719 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.957286 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:51.957944 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:51.958079 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:51.962326 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:50657: BlockingRecv error: recv got EOF from 127.28.104.193:50657 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 06:34:52 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:52 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:52.430457 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:52.421681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:47389)
0619 06:34:52.422205 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:52.422226 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:52.422385 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:52.422814 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:52.422817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:52.423199 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:52.423351 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:52.423360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.424373 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.424377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:52.425755 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:52.425767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.426444 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.426448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:52.426601 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:52.427175 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:52.427193 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:52.430219 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":291,"threads_started":1}
W20260619 06:34:52.430757 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:52.421745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:47389 (local address 127.28.104.254:37635)
0619 06:34:52.422021 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:52.422031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:52.422315 (+ 284us) server_negotiation.cc:407] Connection header received
0619 06:34:52.422621 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:52.422625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:52.422687 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:52.422768 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:52.423477 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.424243 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.425890 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.426309 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.426406 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:52.430476 (+ 4070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:47389: BlockingRecv error: recv got EOF from 127.28.104.194:47389 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:34:52 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:52 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:52.902573 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:52.894546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:60129)
0619 06:34:52.895063 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:52.895085 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:52.895198 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:52.895508 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:52.895512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:52.895896 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:52.896037 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:52.896045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.897178 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.897182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:52.898065 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:52.898075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.898768 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.898772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:52.898884 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:52.899480 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:52.899496 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:52.902355 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":299,"threads_started":1}
W20260619 06:34:52.902870 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:52.894764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:60129 (local address 127.28.104.254:37635)
0619 06:34:52.894930 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:52.894935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:52.895166 (+ 231us) server_negotiation.cc:407] Connection header received
0619 06:34:52.895304 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:52.895308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:52.895376 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:52.895471 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:52.896178 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.896980 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.898200 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.898641 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.898739 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:52.902593 (+ 3854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:60129: BlockingRecv error: recv got EOF from 127.28.104.195:60129 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 06:34:52 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:52 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:52.973052 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:52.963360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:57331)
0619 06:34:52.963829 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:52.963845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:52.963977 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:52.964378 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:52.964383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:52.964862 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:52.965127 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:52.965154 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.966713 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.966717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:52.967748 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:52.967760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.968504 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.968509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:52.968741 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:52.969419 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:52.969444 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:52.972639 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":270,"threads_started":1}
W20260619 06:34:52.973120 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:52.963432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:57331 (local address 127.28.104.254:37635)
0619 06:34:52.963695 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:52.963702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:52.963924 (+ 222us) server_negotiation.cc:407] Connection header received
0619 06:34:52.964110 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:52.964116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:52.964197 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:52.964312 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:52.965294 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.966561 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.967897 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:52.968365 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:52.968487 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:52.972862 (+ 4375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:57331: BlockingRecv error: recv got EOF from 127.28.104.193:57331 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 06:34:53 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:53 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:53.440493 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:53.431543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:46661)
0619 06:34:53.432116 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:53.432141 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:53.432265 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:53.432573 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:53.432577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:53.433083 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:53.433324 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:53.433335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.434359 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.434363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:53.435338 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:53.435353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.436158 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.436162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:53.436351 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:53.437009 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:53.437079 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:53.440148 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":317,"threads_started":1}
W20260619 06:34:53.440603 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:53.431590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:46661 (local address 127.28.104.254:37635)
0619 06:34:53.431759 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:53.431763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:53.432216 (+ 453us) server_negotiation.cc:407] Connection header received
0619 06:34:53.432367 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:53.432370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:53.432431 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:53.432519 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:53.433458 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.434185 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.435464 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.435873 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.435963 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:53.440352 (+ 4389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:46661: BlockingRecv error: recv got EOF from 127.28.104.194:46661 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 06:34:53 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:53 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:53.911831 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:53.903459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:49129)
0619 06:34:53.904067 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:53.904090 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:53.904251 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:53.904594 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:53.904599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:53.905096 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:53.905277 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:53.905286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.906299 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.906306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:53.907201 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:53.907212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.907905 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.907911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:53.908102 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:53.908742 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:53.908760 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:53.911571 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":312,"threads_started":1}
W20260619 06:34:53.911927 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:53.903656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:49129 (local address 127.28.104.254:37635)
0619 06:34:53.903897 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:53.903903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:53.904177 (+ 274us) server_negotiation.cc:407] Connection header received
0619 06:34:53.904375 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:53.904380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:53.904451 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:53.904559 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:53.905430 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.906172 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.907350 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.907750 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.907886 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:53.911756 (+ 3870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:49129: BlockingRecv error: recv got EOF from 127.28.104.195:49129 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 06:34:53 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:53 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:53.983124 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:53.974129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:44495)
0619 06:34:53.974708 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:53.974754 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:53.974914 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:53.975370 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:53.975376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:53.975903 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:53.976149 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:53.976160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.977397 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.977401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:53.978328 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:53.978340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.979107 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.979112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:53.979242 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:53.979885 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:53.979904 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:53.982857 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":381,"threads_started":1}
W20260619 06:34:53.983386 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:53.974183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:44495 (local address 127.28.104.254:37635)
0619 06:34:53.974367 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:53.974373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:53.974856 (+ 483us) server_negotiation.cc:407] Connection header received
0619 06:34:53.975056 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:53.975062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:53.975192 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:53.975308 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:53.976291 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.977269 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.978462 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:53.978975 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:53.979079 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:53.983070 (+ 3991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:44495: BlockingRecv error: recv got EOF from 127.28.104.193:44495 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:34:54 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:54 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:54.450142 5951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:54.441568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:49637)
0619 06:34:54.442087 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:54.442107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:54.442269 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:54.442670 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:54.442675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:54.443105 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:54.443253 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:54.443262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.444234 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.444242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:54.445431 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:54.445443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.446295 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.446299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:54.446440 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:54.447093 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:54.447117 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:54.449909 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
W20260619 06:34:54.450421 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:54.441610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:49637 (local address 127.28.104.254:37635)
0619 06:34:54.441835 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:54.441841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:54.442193 (+ 352us) server_negotiation.cc:407] Connection header received
0619 06:34:54.442401 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:54.442405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:54.442465 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:54.442637 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:54.443391 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.444097 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.445583 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.446147 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.446278 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:54.450149 (+ 3871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:49637: BlockingRecv error: recv got EOF from 127.28.104.194:49637 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 06:34:54 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:54 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:54.922158 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:54.912923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:40061)
0619 06:34:54.913558 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:54.913586 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:54.913721 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:54.914113 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:54.914117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:54.914647 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:54.914878 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:54.914890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.916027 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.916032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:54.917126 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:54.917143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.917896 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.917901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:54.918041 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:54.918862 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:54.918881 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:54.921888 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":409,"threads_started":1}
W20260619 06:34:54.922412 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:54.912981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:40061 (local address 127.28.104.254:37635)
0619 06:34:54.913273 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:54.913283 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:54.913714 (+ 431us) server_negotiation.cc:407] Connection header received
0619 06:34:54.913871 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:54.913874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:54.913942 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:54.914058 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:54.915035 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.915836 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.917307 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.917754 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.917850 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:54.922129 (+ 4279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:40061: BlockingRecv error: recv got EOF from 127.28.104.195:40061 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:34:54 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:54 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:54.993314 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:54.984297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:45943)
0619 06:34:54.984840 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:54.984857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:54.984974 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:54.985322 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:54.985325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:54.985693 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:54.985848 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:54.985856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.987035 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.987044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:54.988411 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:54.988426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.989375 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.989379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:54.989492 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:54.990185 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:54.990202 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:54.993055 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":300,"threads_started":1}
W20260619 06:34:54.993665 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:54.984424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:45943 (local address 127.28.104.254:37635)
0619 06:34:54.984705 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:54.984712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:54.984942 (+ 230us) server_negotiation.cc:407] Connection header received
0619 06:34:54.985103 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:54.985107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:54.985186 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:54.985274 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:54.986056 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.986874 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.988594 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:54.989216 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:54.989324 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:54.993284 (+ 3960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:45943: BlockingRecv error: recv got EOF from 127.28.104.193:45943 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 19 06:34:55 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:55 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:55.459897 5954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:55.451002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:45351)
0619 06:34:55.451512 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:55.451535 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:55.451653 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:55.451954 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:55.451957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:55.452343 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:55.452498 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:55.452508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.453855 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.453859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:55.455202 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:55.455225 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.455924 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.455928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:55.456048 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:55.456718 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:55.456736 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:55.459663 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":233,"threads_started":1}
W20260619 06:34:55.460086 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:55.451242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:45351 (local address 127.28.104.254:37635)
0619 06:34:55.451485 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:55.451490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:55.451619 (+ 129us) server_negotiation.cc:407] Connection header received
0619 06:34:55.451755 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:55.451773 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:55.451832 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:55.451915 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:55.452649 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.453723 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.455348 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.455771 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.455867 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:55.459865 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:45351: BlockingRecv error: recv got EOF from 127.28.104.194:45351 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 06:34:55 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:55 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:55.932301 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:55.923238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:44871)
0619 06:34:55.923856 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:55.923876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:55.923996 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:55.924382 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:55.924386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:55.924776 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:55.924935 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:55.924944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.926273 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.926277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:55.927212 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:55.927224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.928140 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.928149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:55.928302 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:55.929116 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:55.929135 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:55.932058 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":351,"threads_started":1}
W20260619 06:34:55.932540 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:55.923327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:44871 (local address 127.28.104.254:37635)
0619 06:34:55.923583 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:55.923594 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:55.923978 (+ 384us) server_negotiation.cc:407] Connection header received
0619 06:34:55.924133 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:55.924137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:55.924202 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:55.924355 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:55.925390 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.926122 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.927354 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.927900 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.928061 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:55.932296 (+ 4235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:44871: BlockingRecv error: recv got EOF from 127.28.104.195:44871 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 06:34:55 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:55 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:56.003177 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:55.994249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:46917)
0619 06:34:55.994777 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:55.994795 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:55.994922 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:55.995250 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:55.995253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:55.995677 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:55.995942 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:55.995953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.997121 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.997125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:55.998104 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:55.998127 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.998792 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.998796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:55.998931 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:55.999596 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:55.999616 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:56.002814 (+ 3198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":291,"threads_started":1}
W20260619 06:34:56.003306 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:55.994328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:46917 (local address 127.28.104.254:37635)
0619 06:34:55.994558 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:55.994563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:55.994876 (+ 313us) server_negotiation.cc:407] Connection header received
0619 06:34:55.995023 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:55.995027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:55.995099 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:55.995189 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:55.996100 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.996952 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.998241 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:55.998646 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:55.998736 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:56.003022 (+ 4286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:46917: BlockingRecv error: recv got EOF from 127.28.104.193:46917 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Jun 19 06:34:56 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:56 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:56.469751 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:56.460909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:41711)
0619 06:34:56.461447 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:56.461467 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:56.461580 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:56.461916 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:56.461919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:56.462294 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:56.462472 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:56.462481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:56.463578 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:56.463582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:56.464490 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:56.464502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:56.465363 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:56.465367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:56.465492 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:56.466150 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:56.466169 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:56.469450 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":306,"threads_started":1}
W20260619 06:34:56.469830 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:56.460980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:41711 (local address 127.28.104.254:37635)
0619 06:34:56.461205 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:56.461210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:56.461547 (+ 337us) server_negotiation.cc:407] Connection header received
0619 06:34:56.461719 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:56.461723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:56.461787 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:56.461878 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:56.462625 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:56.463452 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:56.464632 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:56.465217 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:56.465316 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:56.469655 (+ 4339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:41711: BlockingRecv error: recv got EOF from 127.28.104.194:41711 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 19 06:34:56 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:56 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.195@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:56.946146 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:56.933290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.195:42039)
0619 06:34:56.933771 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:56.933792 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:56.933941 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:56.934273 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:56.934277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:56.934684 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:56.934865 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:56.934874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:56.935867 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:56.935873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:56.936883 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:56.936895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:56.937722 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:56.937728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:56.937903 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:56.942482 (+ 4579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:56.942502 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:56.945887 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":246,"threads_started":1}
W20260619 06:34:56.946305 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:56.933362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:42039 (local address 127.28.104.254:37635)
0619 06:34:56.933602 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:56.933610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:56.933873 (+ 263us) server_negotiation.cc:407] Connection header received
0619 06:34:56.934066 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:56.934073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:56.934141 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:56.934244 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:56.935005 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:56.935718 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:56.937051 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:56.937564 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:56.937709 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:56.946113 (+ 8404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.195:42039: BlockingRecv error: recv got EOF from 127.28.104.195:42039 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 06:34:57 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:57 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.193@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:57.013603 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:57.004543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.193:35123)
0619 06:34:57.005161 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:57.005182 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:57.005337 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:57.005762 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:57.005768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:57.006290 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:57.006498 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:57.006507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.007687 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.007691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:57.008736 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:57.008749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.009522 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.009527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:57.009693 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:57.010344 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:57.010360 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:57.013202 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":305,"threads_started":1}
W20260619 06:34:57.013659 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:57.004661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:35123 (local address 127.28.104.254:37635)
0619 06:34:57.004856 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:57.004862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:57.005275 (+ 413us) server_negotiation.cc:407] Connection header received
0619 06:34:57.005468 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:57.005473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:57.005601 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:57.005704 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:57.006640 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.007555 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.008897 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.009383 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.009498 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:57.013426 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.193:35123: BlockingRecv error: recv got EOF from 127.28.104.193:35123 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 06:34:57 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:34:57 dist-test-slave-nf4n krb5kdc[4906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.104.194@KRBTEST.COM for oryx/127.28.104.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:34:57.479871 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:57.471033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37635 (local address 127.28.104.194:42919)
0619 06:34:57.471645 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:57.471683 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:57.471853 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:57.472216 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:57.472221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:57.472761 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:57.472924 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:57.472933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.474188 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.474196 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:57.475127 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:57.475139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.475870 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.475876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:57.476056 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:57.476661 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:34:57.476678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:57.479477 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.104.254:37635: Server oryx/127.28.104.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":335,"threads_started":1}
W20260619 06:34:57.479921 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:34:57.471099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:42919 (local address 127.28.104.254:37635)
0619 06:34:57.471306 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:57.471315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:57.471788 (+ 473us) server_negotiation.cc:407] Connection header received
0619 06:34:57.471992 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:57.471997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:57.472072 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:57.472180 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:57.473109 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.474006 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.475273 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.475722 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.475859 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:57.479683 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.104.194:42919: BlockingRecv error: recv got EOF from 127.28.104.194:42919 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:34:57.769217 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5498
I20260619 06:34:57.786808 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:57.770453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.28.104.193:37651)
0619 06:34:57.771115 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:57.771123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:57.771153 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:34:57.771222 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:57.771229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:57.771483 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:57.771662 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:57.773163 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.774491 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.776608 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.777737 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.777890 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:57.782428 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:57.782442 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:57.782463 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:57.782532 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:57.786164 (+ 3632us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.104.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 06:34:57.786476 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47180: Request ticket server kudu/127.28.104.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":339,"threads_started":1}
W20260619 06:34:57.786895 5962 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47180: Request ticket server kudu/127.28.104.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 06:34:57.787199 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5498: Not authorized: Client connection negotiation failed: client connection to 127.28.104.193:37651: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.104.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 06:34:57.787416 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 5498
I20260619 06:34:57.809594 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5651
I20260619 06:34:57.822248 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:57.810637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.28.104.194:39975)
0619 06:34:57.811221 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:57.811228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:57.811259 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:34:57.811324 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:57.811330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:57.811551 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:57.811780 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:57.813142 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.814273 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.815489 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.816078 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.816227 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:57.818904 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:57.818914 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:57.818930 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:57.818964 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:57.821497 (+ 2533us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.104.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 06:34:57.821868 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51950: Request ticket server kudu/127.28.104.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":272,"threads_started":1}
W20260619 06:34:57.822335 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5651: Not authorized: Client connection negotiation failed: client connection to 127.28.104.194:39975: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.104.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 06:34:57.822381 5963 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51950: Request ticket server kudu/127.28.104.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 06:34:57.822501 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 5651
I20260619 06:34:57.839522 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5783
I20260619 06:34:57.852310 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:57.840545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.28.104.195:46575)
0619 06:34:57.841311 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:57.841319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:57.841346 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:34:57.841415 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:57.841423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:57.841669 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:57.841832 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:57.843178 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.844277 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.845731 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:57.846330 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:57.846451 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:57.848962 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:57.848970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:57.848985 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:34:57.849109 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:57.851776 (+ 2667us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.104.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 06:34:57.852044 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50238: Request ticket server kudu/127.28.104.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":355,"threads_started":1}
W20260619 06:34:57.852427 5964 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50238: Request ticket server kudu/127.28.104.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 06:34:57.852470 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5783: Not authorized: Client connection negotiation failed: client connection to 127.28.104.195:46575: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.104.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 06:34:57.852618 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 5783
I20260619 06:34:57.869309 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5428
I20260619 06:34:57.870715 5491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:34:57.989811 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 5428
2026-06-19T06:34:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21200 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T06:34:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:34:58Z Disabled control of system clock
I20260619 06:34:58.069161 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:40743
--webserver_interface=127.28.104.254
--webserver_port=0
--builtin_ntp_servers=127.28.104.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:40743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:34:58.401347 5969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:58.401651 5969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:58.401733 5969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:58.409094 5969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:34:58.409219 5969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:58.409256 5969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:58.409314 5969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:34:58.409358 5969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:34:58.418756 5969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:40743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:40743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:58.420116 5969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:58.422048 5969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:58.434453 5977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:58.434616 5974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:58.434537 5975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:58.434871 5969 server_base.cc:1034] running on GCE node
I20260619 06:34:58.435945 5969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:58.437495 5969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:58.438701 5969 hybrid_clock.cc:648] HybridClock initialized: now 1781850898438659 us; error 34 us; skew 500 ppm
I20260619 06:34:58.439065 5969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:58.441342 5969 webserver.cc:460] Webserver started at http://127.28.104.254:39001/ using document root <none> and password file <none>
I20260619 06:34:58.442052 5969 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:58.442183 5969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:58.442510 5969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:58.446442 5969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "aea1723c7eca40d29851872582a69174"
format_stamp: "Formatted at 2026-06-19 06:34:58 on dist-test-slave-nf4n"
server_key: "20b3bfddd59a3e48bd77346280550fc3"
server_key_iv: "6249acc9d58dfeabf5b6d028c4a048be"
server_key_version: "encryptionkey@0"
I20260619 06:34:58.447468 5969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "aea1723c7eca40d29851872582a69174"
format_stamp: "Formatted at 2026-06-19 06:34:58 on dist-test-slave-nf4n"
server_key: "20b3bfddd59a3e48bd77346280550fc3"
server_key_iv: "6249acc9d58dfeabf5b6d028c4a048be"
server_key_version: "encryptionkey@0"
I20260619 06:34:58.453208 5969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 06:34:58.457150 5983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:58.458628 5969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:34:58.458850 5969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "aea1723c7eca40d29851872582a69174"
format_stamp: "Formatted at 2026-06-19 06:34:58 on dist-test-slave-nf4n"
server_key: "20b3bfddd59a3e48bd77346280550fc3"
server_key_iv: "6249acc9d58dfeabf5b6d028c4a048be"
server_key_version: "encryptionkey@0"
I20260619 06:34:58.459110 5969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:58.483501 5969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:58.484503 5969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:58.484850 5969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:58.505676 5969 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:40743
I20260619 06:34:58.505719 6034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:40743 every 8 connection(s)
I20260619 06:34:58.507225 5969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:34:58.511255 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:34:58.513190 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 5969
I20260619 06:34:58.513461 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:34:58.514155 29091 external_mini_cluster.cc:1428] Setting key 0a9995f7ffb01462975d1e48aa7f25e9
I20260619 06:34:58.524326 6035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174: Bootstrap starting.
I20260619 06:34:58.528427 6035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174: Neither blocks nor log segments found. Creating new log.
I20260619 06:34:58.529687 6035 log.cc:824] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174: Log is configured to *not* fsync() on all Append() calls
I20260619 06:34:58.530121 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:58.516540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.28.104.254:40743)
0619 06:34:58.517822 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:58.517835 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:58.517887 (+ 52us) server_negotiation.cc:407] Connection header received
0619 06:34:58.519728 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:58.519755 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:58.520329 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:58.520788 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:58.521693 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:58.523039 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:58.524942 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:58.525762 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:58.525911 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:58.527338 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:58.527346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:58.527362 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:58.527951 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:58.528045 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:58.528066 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:58.528157 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:58.528564 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 06:34:58.528928 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":502,"threads_started":1}
I20260619 06:34:58.534147 6035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174: No bootstrap required, opened a new log
I20260619 06:34:58.542382 6035 raft_consensus.cc:348] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea1723c7eca40d29851872582a69174" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40743 } }
I20260619 06:34:58.542832 6035 raft_consensus.cc:374] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:34:58.542953 6035 raft_consensus.cc:729] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea1723c7eca40d29851872582a69174, State: Initialized, Role: FOLLOWER
I20260619 06:34:58.543624 6035 consensus_queue.cc:260] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea1723c7eca40d29851872582a69174" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40743 } }
I20260619 06:34:58.543872 6035 raft_consensus.cc:388] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:34:58.544021 6035 raft_consensus.cc:482] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:34:58.544205 6035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:34:58.547086 6035 raft_consensus.cc:504] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea1723c7eca40d29851872582a69174" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40743 } }
I20260619 06:34:58.547642 6035 leader_election.cc:302] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aea1723c7eca40d29851872582a69174; no voters:
I20260619 06:34:58.548270 6035 leader_election.cc:288] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:34:58.548445 6040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:34:58.549281 6040 raft_consensus.cc:686] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [term 1 LEADER]: Becoming Leader. State: Replica: aea1723c7eca40d29851872582a69174, State: Running, Role: LEADER
I20260619 06:34:58.549939 6040 consensus_queue.cc:237] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea1723c7eca40d29851872582a69174" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40743 } }
I20260619 06:34:58.550591 6035 sys_catalog.cc:564] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:34:58.553207 6042 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aea1723c7eca40d29851872582a69174. Latest consensus state: current_term: 1 leader_uuid: "aea1723c7eca40d29851872582a69174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea1723c7eca40d29851872582a69174" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40743 } } }
I20260619 06:34:58.553306 6041 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aea1723c7eca40d29851872582a69174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea1723c7eca40d29851872582a69174" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40743 } } }
I20260619 06:34:58.553654 6042 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:58.553678 6041 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174 [sys.catalog]: This master's current role is: LEADER
I20260619 06:34:58.555365 6045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:34:58.560786 6045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:34:58.570816 6045 catalog_manager.cc:1347] Generated new cluster ID: f7dfb0b83a7c41fd8e914f7fb604a521
I20260619 06:34:58.570941 6045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:34:58.594233 6045 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:34:58.596129 6045 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:34:58.602170 6045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aea1723c7eca40d29851872582a69174: Generated new TSK 0
I20260619 06:34:58.603149 6045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:34:58.619895 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40743
--builtin_ntp_servers=127.28.104.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 06:34:58.966063 6059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:58.966380 6059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:58.966534 6059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:58.974350 6059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:58.974469 6059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:58.974655 6059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:34:58.984814 6059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:58.986310 6059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:58.988191 6059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:59.003605 6066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:59.003571 6065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:59.004161 6068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:59.004623 6059 server_base.cc:1034] running on GCE node
I20260619 06:34:59.005431 6059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:59.006644 6059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:59.007918 6059 hybrid_clock.cc:648] HybridClock initialized: now 1781850899007862 us; error 63 us; skew 500 ppm
I20260619 06:34:59.008294 6059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:59.010987 6059 webserver.cc:460] Webserver started at http://127.28.104.193:43013/ using document root <none> and password file <none>
I20260619 06:34:59.011806 6059 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:59.011950 6059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:59.012292 6059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:59.015381 6059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "13bd896dc4274574ba4da8d800680dac"
format_stamp: "Formatted at 2026-06-19 06:34:59 on dist-test-slave-nf4n"
server_key: "c6070698b5877b5953fcd17288b0ddaf"
server_key_iv: "9052f5dbceff2b6e9c50f1ec59f2c058"
server_key_version: "encryptionkey@0"
I20260619 06:34:59.016218 6059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "13bd896dc4274574ba4da8d800680dac"
format_stamp: "Formatted at 2026-06-19 06:34:59 on dist-test-slave-nf4n"
server_key: "c6070698b5877b5953fcd17288b0ddaf"
server_key_iv: "9052f5dbceff2b6e9c50f1ec59f2c058"
server_key_version: "encryptionkey@0"
I20260619 06:34:59.021999 6059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 06:34:59.025732 6074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:59.027366 6059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:34:59.027611 6059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "13bd896dc4274574ba4da8d800680dac"
format_stamp: "Formatted at 2026-06-19 06:34:59 on dist-test-slave-nf4n"
server_key: "c6070698b5877b5953fcd17288b0ddaf"
server_key_iv: "9052f5dbceff2b6e9c50f1ec59f2c058"
server_key_version: "encryptionkey@0"
I20260619 06:34:59.027870 6059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:59.043577 6059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:59.044566 6059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:59.044885 6059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:59.045971 6059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:59.047479 6059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:59.047600 6059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:59.047737 6059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:59.047828 6059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:59.081537 6059 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:34571
I20260619 06:34:59.081593 6186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:34571 every 8 connection(s)
I20260619 06:34:59.082976 6059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:34:59.083767 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 6059
I20260619 06:34:59.084012 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:34:59.084885 29091 external_mini_cluster.cc:1428] Setting key ec2d2cb29fad517379d6fb58a29af785
I20260619 06:34:59.088001 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40743
--builtin_ntp_servers=127.28.104.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:34:59.098080 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:59.086267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:57601 (local address 127.28.104.254:40743)
0619 06:34:59.086753 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:59.086760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:59.088496 (+ 1736us) server_negotiation.cc:407] Connection header received
0619 06:34:59.090345 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:59.090353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:59.090445 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:59.090566 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:59.091716 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:59.092900 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:59.094994 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:59.095408 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:59.095515 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:59.097263 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:59.097269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:59.097272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:59.097293 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:59.097338 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:59.097351 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:59.097435 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:59.097727 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:34:59.097815 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
I20260619 06:34:59.099776 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:59.087171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40743 (local address 127.28.104.193:57601)
0619 06:34:59.088311 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:59.088366 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:59.090072 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:59.090831 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:59.090840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:59.090934 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:59.091529 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:59.091556 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:59.093122 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:59.093126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:59.094827 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:59.094845 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:59.095658 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:59.095667 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:59.096018 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:59.096841 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:34:59.096847 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:59.096905 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:34:59.097051 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:59.097456 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:59.097468 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:59.097479 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:34:59.097687 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 06:34:59.098176 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":359,"threads_started":1}
I20260619 06:34:59.101550 6187 heartbeater.cc:347] Connected to a master server at 127.28.104.254:40743
I20260619 06:34:59.101888 6187 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:59.102907 6187 heartbeater.cc:511] Master 127.28.104.254:40743 requested a full tablet report, sending...
I20260619 06:34:59.105333 6000 ts_manager.cc:194] Registered new tserver with Master: 13bd896dc4274574ba4da8d800680dac (127.28.104.193:34571)
I20260619 06:34:59.106906 6000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.193:57601
W20260619 06:34:59.433521 6191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:59.433842 6191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:59.434020 6191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:59.441921 6191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:59.442047 6191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:59.442222 6191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:34:59.453491 6191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:59.454936 6191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:59.456866 6191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:59.471427 6197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:59.471606 6196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:59.471583 6199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:59.472563 6191 server_base.cc:1034] running on GCE node
I20260619 06:34:59.473322 6191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:59.474452 6191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:59.475723 6191 hybrid_clock.cc:648] HybridClock initialized: now 1781850899475672 us; error 68 us; skew 500 ppm
I20260619 06:34:59.476078 6191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:59.478503 6191 webserver.cc:460] Webserver started at http://127.28.104.194:46463/ using document root <none> and password file <none>
I20260619 06:34:59.479249 6191 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:59.479380 6191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:59.479713 6191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:59.482671 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "52af9971c0284e5ba577292a68abd7e5"
format_stamp: "Formatted at 2026-06-19 06:34:59 on dist-test-slave-nf4n"
server_key: "a66db4da94043c20b92e19eef832a508"
server_key_iv: "264e7380f7ea6606907bc9b860d7d855"
server_key_version: "encryptionkey@0"
I20260619 06:34:59.483443 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "52af9971c0284e5ba577292a68abd7e5"
format_stamp: "Formatted at 2026-06-19 06:34:59 on dist-test-slave-nf4n"
server_key: "a66db4da94043c20b92e19eef832a508"
server_key_iv: "264e7380f7ea6606907bc9b860d7d855"
server_key_version: "encryptionkey@0"
I20260619 06:34:59.488849 6191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:34:59.492475 6205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:59.493939 6191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:34:59.494124 6191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "52af9971c0284e5ba577292a68abd7e5"
format_stamp: "Formatted at 2026-06-19 06:34:59 on dist-test-slave-nf4n"
server_key: "a66db4da94043c20b92e19eef832a508"
server_key_iv: "264e7380f7ea6606907bc9b860d7d855"
server_key_version: "encryptionkey@0"
I20260619 06:34:59.494359 6191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:34:59.542666 6191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:34:59.543560 6191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:34:59.543834 6191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:34:59.544796 6191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:34:59.546451 6191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:34:59.546542 6191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:59.546620 6191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:34:59.546686 6191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:59.580574 6191 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:46387
I20260619 06:34:59.580698 6318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:46387 every 8 connection(s)
I20260619 06:34:59.581940 6191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:34:59.586463 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 6191
I20260619 06:34:59.586652 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:34:59.587103 29091 external_mini_cluster.cc:1428] Setting key 8c479ef0be2e160a930433c4d2188f22
I20260619 06:34:59.590363 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40743
--builtin_ntp_servers=127.28.104.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:34:59.595930 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:59.584564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:48665 (local address 127.28.104.254:40743)
0619 06:34:59.584767 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:34:59.584772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:34:59.586243 (+ 1471us) server_negotiation.cc:407] Connection header received
0619 06:34:59.588412 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:34:59.588416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:34:59.588480 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:34:59.588571 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:34:59.589954 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:59.591073 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:59.592651 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:34:59.593354 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:34:59.593492 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:59.594951 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:34:59.594960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:34:59.594967 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:34:59.594993 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:34:59.595048 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:34:59.595066 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:34:59.595162 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:34:59.595402 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 06:34:59.595511 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":81}
I20260619 06:34:59.596877 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:34:59.585148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40743 (local address 127.28.104.194:48665)
0619 06:34:59.586057 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0619 06:34:59.586107 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:34:59.588102 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:34:59.588884 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:34:59.588897 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:34:59.589076 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:34:59.589781 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:59.589799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:59.591214 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:59.591219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:59.592492 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:34:59.592505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:34:59.593598 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:34:59.593604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:34:59.593812 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:34:59.594624 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:34:59.594627 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:34:59.594670 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:34:59.594752 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:34:59.595173 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:34:59.595180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:34:59.595186 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:34:59.595358 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 06:34:59.595723 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":271,"threads_started":1}
I20260619 06:34:59.598441 6319 heartbeater.cc:347] Connected to a master server at 127.28.104.254:40743
I20260619 06:34:59.598825 6319 heartbeater.cc:464] Registering TS with master...
I20260619 06:34:59.599659 6319 heartbeater.cc:511] Master 127.28.104.254:40743 requested a full tablet report, sending...
I20260619 06:34:59.601183 6000 ts_manager.cc:194] Registered new tserver with Master: 52af9971c0284e5ba577292a68abd7e5 (127.28.104.194:46387)
I20260619 06:34:59.601989 6000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.194:48665
W20260619 06:34:59.927548 6322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:34:59.927898 6322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:34:59.928071 6322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:34:59.935531 6322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:34:59.935688 6322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:34:59.935875 6322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:34:59.945132 6322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:34:59.946516 6322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:34:59.948393 6322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:34:59.962416 6327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:59.962493 6330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:34:59.962513 6328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:34:59.966789 6322 server_base.cc:1034] running on GCE node
I20260619 06:34:59.967432 6322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:34:59.968474 6322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:34:59.969712 6322 hybrid_clock.cc:648] HybridClock initialized: now 1781850899969657 us; error 45 us; skew 500 ppm
I20260619 06:34:59.970044 6322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:34:59.972498 6322 webserver.cc:460] Webserver started at http://127.28.104.195:40707/ using document root <none> and password file <none>
I20260619 06:34:59.973300 6322 fs_manager.cc:359] Metadata directory not provided
I20260619 06:34:59.973392 6322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:34:59.973688 6322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:34:59.976853 6322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "3693a436c120460ca507b2f6741d547d"
format_stamp: "Formatted at 2026-06-19 06:34:59 on dist-test-slave-nf4n"
server_key: "4244d2179ddb89d91331794c78e28240"
server_key_iv: "a611f6e6601c2b527edc47fa93bc271b"
server_key_version: "encryptionkey@0"
I20260619 06:34:59.977751 6322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "3693a436c120460ca507b2f6741d547d"
format_stamp: "Formatted at 2026-06-19 06:34:59 on dist-test-slave-nf4n"
server_key: "4244d2179ddb89d91331794c78e28240"
server_key_iv: "a611f6e6601c2b527edc47fa93bc271b"
server_key_version: "encryptionkey@0"
I20260619 06:34:59.983057 6322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 06:34:59.986742 6337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:34:59.988271 6322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:34:59.988474 6322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "3693a436c120460ca507b2f6741d547d"
format_stamp: "Formatted at 2026-06-19 06:34:59 on dist-test-slave-nf4n"
server_key: "4244d2179ddb89d91331794c78e28240"
server_key_iv: "a611f6e6601c2b527edc47fa93bc271b"
server_key_version: "encryptionkey@0"
I20260619 06:34:59.988727 6322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:00.026801 6322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:00.027730 6322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:00.028043 6322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:00.029145 6322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:35:00.030579 6322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:35:00.030689 6322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:00.030817 6322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:35:00.030906 6322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:00.064986 6322 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:36347
I20260619 06:35:00.065092 6450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:36347 every 8 connection(s)
I20260619 06:35:00.066483 6322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:35:00.076263 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 6322
I20260619 06:35:00.076467 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:35:00.076938 29091 external_mini_cluster.cc:1428] Setting key 686ef83db7f1a3f3391b536652c8a86a
I20260619 06:35:00.080541 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:00.069473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:36863 (local address 127.28.104.254:40743)
0619 06:35:00.069721 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:00.069725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:00.071412 (+ 1687us) server_negotiation.cc:407] Connection header received
0619 06:35:00.073187 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:00.073192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:00.073277 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:00.073364 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:00.074697 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:00.075425 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:00.077361 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:00.077926 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:00.078056 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:00.079598 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:00.079605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:00.079609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:00.079633 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:00.079677 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:00.079723 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:00.079831 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:00.080187 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 06:35:00.080285 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260619 06:35:00.081851 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:00.070104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40743 (local address 127.28.104.195:36863)
0619 06:35:00.071176 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:00.071252 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:00.072850 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:00.073641 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:00.073652 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:00.073753 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:35:00.074451 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:00.074472 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:00.075616 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:00.075621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:00.077214 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:00.077227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:00.078167 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:00.078174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:00.078512 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:00.079266 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:35:00.079268 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:35:00.079310 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:35:00.079400 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:35:00.079889 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:35:00.079898 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:35:00.079905 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:35:00.080169 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 06:35:00.080687 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":293,"threads_started":1}
I20260619 06:35:00.083238 6451 heartbeater.cc:347] Connected to a master server at 127.28.104.254:40743
I20260619 06:35:00.083578 6451 heartbeater.cc:464] Registering TS with master...
I20260619 06:35:00.084503 6451 heartbeater.cc:511] Master 127.28.104.254:40743 requested a full tablet report, sending...
I20260619 06:35:00.086133 6000 ts_manager.cc:194] Registered new tserver with Master: 3693a436c120460ca507b2f6741d547d (127.28.104.195:36347)
I20260619 06:35:00.086982 6000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.195:36863
I20260619 06:35:00.092725 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:35:00.099229 6189 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:35:00.097376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.28.104.254:40743)
0619 06:35:00.097614 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:00.097618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:00.098139 (+ 521us) server_negotiation.cc:407] Connection header received
0619 06:35:00.098360 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:00.098364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:00.098438 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:00.098529 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:00.099023 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50000: BlockingRecv error: recv got EOF from 127.0.0.1:50000 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 06:35:00.099956 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 6059
I20260619 06:35:00.107836 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:00.100868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.28.104.193:34571)
0619 06:35:00.101457 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:00.101463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:00.101490 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:35:00.101582 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:00.101589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:00.101813 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:00.101939 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:00.102264 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:00.103313 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:00.104570 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:00.105228 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:00.105348 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:00.106453 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:00.106461 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:00.106475 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:00.107092 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:00.107141 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:00.107169 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:00.107260 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:00.107404 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 06:35:00.107552 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":314,"threads_started":1}
I20260619 06:35:00.109160 6182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:00.190281 6187 heartbeater.cc:503] Master 127.28.104.254:40743 was elected leader, sending a full tablet report...
I20260619 06:35:00.191313 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 6059
W20260619 06:35:00.207818 5984 connection.cc:535] server connection from 127.28.104.193:57601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:35:00.207988 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 6191
I20260619 06:35:00.215981 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:00.208911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.28.104.194:46387)
0619 06:35:00.209510 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:00.209520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:00.209551 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:35:00.209648 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:00.209654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:00.209888 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:00.210030 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:00.210407 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:00.211447 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:00.212659 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:00.213308 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:00.213406 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:00.214494 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:00.214506 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:00.214526 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:00.215195 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:00.215252 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:00.215274 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:00.215378 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:00.215538 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 06:35:00.215674 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":340,"threads_started":1}
I20260619 06:35:00.217356 6314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:00.301287 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 6191
I20260619 06:35:00.317231 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 6322
I20260619 06:35:00.325596 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:00.318226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.28.104.195:36347)
0619 06:35:00.318880 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:00.318887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:00.318921 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:35:00.318989 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:00.318995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:00.319227 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:00.319374 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:00.319762 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:00.320916 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:00.322199 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:00.322866 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:00.322976 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:00.324052 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:00.324061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:00.324077 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:00.324768 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:00.324836 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:00.324858 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:00.324953 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:00.325145 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 06:35:00.325268 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":306,"threads_started":1}
I20260619 06:35:00.326961 6446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:00.409067 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 6322
I20260619 06:35:00.427243 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 5969
I20260619 06:35:00.428478 6030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:00.536152 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 5969
2026-06-19T06:35:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2541 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T06:35:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:35:00Z Disabled control of system clock
I20260619 06:35:00.608891 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:41483
--webserver_interface=127.28.104.254
--webserver_port=0
--builtin_ntp_servers=127.28.104.212:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:41483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:35:00.938812 6470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:00.939141 6470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:00.939354 6470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:00.947583 6470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:35:00.947714 6470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:00.947803 6470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:00.947891 6470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:35:00.947970 6470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:35:00.957428 6470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:41483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:41483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:00.958832 6470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:00.960670 6470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:00.972664 6475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:00.972630 6478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:00.972873 6476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:00.974475 6470 server_base.cc:1034] running on GCE node
I20260619 06:35:00.975325 6470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:00.976851 6470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:00.978093 6470 hybrid_clock.cc:648] HybridClock initialized: now 1781850900977980 us; error 103 us; skew 500 ppm
I20260619 06:35:00.978471 6470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:35:00.980993 6470 webserver.cc:460] Webserver started at http://127.28.104.254:36167/ using document root <none> and password file <none>
I20260619 06:35:00.981813 6470 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:00.981958 6470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:00.982303 6470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:35:00.987110 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "9786d6fcb5d24045b502d534eb4ca5cb"
format_stamp: "Formatted at 2026-06-19 06:35:00 on dist-test-slave-nf4n"
server_key: "c1e95ff82eb1942100a59c33b899cda4"
server_key_iv: "b8ed0108d6433b87f97e2e81ff2332fe"
server_key_version: "encryptionkey@0"
I20260619 06:35:00.988054 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "9786d6fcb5d24045b502d534eb4ca5cb"
format_stamp: "Formatted at 2026-06-19 06:35:00 on dist-test-slave-nf4n"
server_key: "c1e95ff82eb1942100a59c33b899cda4"
server_key_iv: "b8ed0108d6433b87f97e2e81ff2332fe"
server_key_version: "encryptionkey@0"
I20260619 06:35:00.993633 6470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:35:00.997272 6484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:00.998646 6470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:35:00.998870 6470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "9786d6fcb5d24045b502d534eb4ca5cb"
format_stamp: "Formatted at 2026-06-19 06:35:00 on dist-test-slave-nf4n"
server_key: "c1e95ff82eb1942100a59c33b899cda4"
server_key_iv: "b8ed0108d6433b87f97e2e81ff2332fe"
server_key_version: "encryptionkey@0"
I20260619 06:35:00.999111 6470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:01.005295 6470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:01.006147 6470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:01.006464 6470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:01.025309 6470 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:41483
I20260619 06:35:01.025342 6536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:41483 every 8 connection(s)
I20260619 06:35:01.026749 6470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:35:01.030596 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 6470
I20260619 06:35:01.030740 6537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:35:01.030859 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:35:01.031520 29091 external_mini_cluster.cc:1428] Setting key ebc375d2049bbe0b2a8fb61992b3e78e
I20260619 06:35:01.041008 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:01.033414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.28.104.254:41483)
0619 06:35:01.034527 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:01.034538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:01.034581 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:35:01.037119 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:01.037147 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:01.037665 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:01.038057 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:01.039253 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:01.039260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:01.039275 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:01.039290 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:01.039361 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:01.039384 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:01.039572 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:01.039690 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 06:35:01.040017 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":418,"threads_started":1}
I20260619 06:35:01.044704 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb: Bootstrap starting.
I20260619 06:35:01.047772 6537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb: Neither blocks nor log segments found. Creating new log.
I20260619 06:35:01.048799 6537 log.cc:824] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb: Log is configured to *not* fsync() on all Append() calls
I20260619 06:35:01.051878 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb: No bootstrap required, opened a new log
I20260619 06:35:01.059594 6537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9786d6fcb5d24045b502d534eb4ca5cb" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41483 } }
I20260619 06:35:01.059979 6537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:35:01.060132 6537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9786d6fcb5d24045b502d534eb4ca5cb, State: Initialized, Role: FOLLOWER
I20260619 06:35:01.060702 6537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9786d6fcb5d24045b502d534eb4ca5cb" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41483 } }
I20260619 06:35:01.060935 6537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:35:01.061111 6537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:35:01.061319 6537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:35:01.064234 6537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9786d6fcb5d24045b502d534eb4ca5cb" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41483 } }
I20260619 06:35:01.064780 6537 leader_election.cc:302] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9786d6fcb5d24045b502d534eb4ca5cb; no voters:
I20260619 06:35:01.065495 6537 leader_election.cc:288] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:35:01.065639 6542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:35:01.066411 6542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [term 1 LEADER]: Becoming Leader. State: Replica: 9786d6fcb5d24045b502d534eb4ca5cb, State: Running, Role: LEADER
I20260619 06:35:01.067068 6542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9786d6fcb5d24045b502d534eb4ca5cb" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41483 } }
I20260619 06:35:01.067755 6537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:35:01.070289 6544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9786d6fcb5d24045b502d534eb4ca5cb. Latest consensus state: current_term: 1 leader_uuid: "9786d6fcb5d24045b502d534eb4ca5cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9786d6fcb5d24045b502d534eb4ca5cb" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41483 } } }
I20260619 06:35:01.070295 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9786d6fcb5d24045b502d534eb4ca5cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9786d6fcb5d24045b502d534eb4ca5cb" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41483 } } }
I20260619 06:35:01.070753 6544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [sys.catalog]: This master's current role is: LEADER
I20260619 06:35:01.070762 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb [sys.catalog]: This master's current role is: LEADER
I20260619 06:35:01.072636 6549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:35:01.078181 6549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:35:01.087610 6549 catalog_manager.cc:1347] Generated new cluster ID: 24d205871c1146f395d9b6c5bd4db9e3
I20260619 06:35:01.087775 6549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:35:01.120771 6549 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:35:01.121351 6549 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:35:01.131893 6549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9786d6fcb5d24045b502d534eb4ca5cb: Generated new TSK 0
I20260619 06:35:01.132781 6549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:35:01.141176 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:41483
--builtin_ntp_servers=127.28.104.212:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:35:01.498113 6561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:01.498390 6561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:01.498601 6561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:01.506196 6561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:01.506301 6561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:01.506465 6561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:35:01.519368 6561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:41483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:01.520780 6561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:01.522622 6561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:01.535964 6567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:01.535919 6569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:01.536012 6566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:01.542470 6561 server_base.cc:1034] running on GCE node
I20260619 06:35:01.543129 6561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:01.544191 6561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:01.545397 6561 hybrid_clock.cc:648] HybridClock initialized: now 1781850901545301 us; error 99 us; skew 500 ppm
I20260619 06:35:01.545753 6561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:35:01.548135 6561 webserver.cc:460] Webserver started at http://127.28.104.193:44677/ using document root <none> and password file <none>
I20260619 06:35:01.548861 6561 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:01.548976 6561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:01.549391 6561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:35:01.552333 6561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "0ed4ed897de74850837798b315a6b96e"
format_stamp: "Formatted at 2026-06-19 06:35:01 on dist-test-slave-nf4n"
server_key: "94faa9f0c054b3fe3b12e1930cbefd72"
server_key_iv: "2d5a099024fec8de1ca898f08df87f56"
server_key_version: "encryptionkey@0"
I20260619 06:35:01.553237 6561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "0ed4ed897de74850837798b315a6b96e"
format_stamp: "Formatted at 2026-06-19 06:35:01 on dist-test-slave-nf4n"
server_key: "94faa9f0c054b3fe3b12e1930cbefd72"
server_key_iv: "2d5a099024fec8de1ca898f08df87f56"
server_key_version: "encryptionkey@0"
I20260619 06:35:01.558490 6561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 06:35:01.562018 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:01.563437 6561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:35:01.563623 6561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "0ed4ed897de74850837798b315a6b96e"
format_stamp: "Formatted at 2026-06-19 06:35:01 on dist-test-slave-nf4n"
server_key: "94faa9f0c054b3fe3b12e1930cbefd72"
server_key_iv: "2d5a099024fec8de1ca898f08df87f56"
server_key_version: "encryptionkey@0"
I20260619 06:35:01.563866 6561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:01.570025 6561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:01.570851 6561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:01.571126 6561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:01.572041 6561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:35:01.573922 6561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:35:01.574035 6561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:01.574143 6561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:35:01.574216 6561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:01.607172 6561 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:36259
I20260619 06:35:01.607228 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:36259 every 8 connection(s)
I20260619 06:35:01.608566 6561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:35:01.617468 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:01.611579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:38737 (local address 127.28.104.254:41483)
0619 06:35:01.612029 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:01.612034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:01.613295 (+ 1261us) server_negotiation.cc:407] Connection header received
0619 06:35:01.615004 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:01.615010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:01.615085 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:01.615209 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:01.616597 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:01.616609 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:01.616613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:01.616630 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:01.616691 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:01.616703 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:01.616839 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:01.617098 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 06:35:01.617169 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":242,"threads_started":1}
I20260619 06:35:01.618177 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 6561
I20260619 06:35:01.618367 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:35:01.618723 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:01.612188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:41483 (local address 127.28.104.193:38737)
0619 06:35:01.613130 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:01.613174 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:01.614738 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:01.615459 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:01.615466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:01.615559 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:35:01.616310 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:35:01.616312 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:35:01.616351 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:35:01.616434 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:35:01.616800 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:35:01.616810 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:35:01.616818 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:35:01.617198 (+ 380us) client_negotiation.cc:240] Negotiation successful
0619 06:35:01.617456 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":256,"threads_started":1}
I20260619 06:35:01.619187 29091 external_mini_cluster.cc:1428] Setting key bed083daea7e99d41138cbb92694d758
I20260619 06:35:01.620340 6689 heartbeater.cc:347] Connected to a master server at 127.28.104.254:41483
I20260619 06:35:01.620646 6689 heartbeater.cc:464] Registering TS with master...
I20260619 06:35:01.621431 6689 heartbeater.cc:511] Master 127.28.104.254:41483 requested a full tablet report, sending...
I20260619 06:35:01.622195 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:41483
--builtin_ntp_servers=127.28.104.212:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 06:35:01.623406 6501 ts_manager.cc:194] Registered new tserver with Master: 0ed4ed897de74850837798b315a6b96e (127.28.104.193:36259)
W20260619 06:35:01.951766 6693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:01.952065 6693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:01.952219 6693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:01.959827 6693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:01.959949 6693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:01.960158 6693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:35:01.969996 6693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:41483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:01.971343 6693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:01.973413 6693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:01.986996 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:01.987025 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:01.986997 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:01.988334 6693 server_base.cc:1034] running on GCE node
I20260619 06:35:01.989113 6693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:01.990110 6693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:01.991382 6693 hybrid_clock.cc:648] HybridClock initialized: now 1781850901991328 us; error 40 us; skew 500 ppm
I20260619 06:35:01.991796 6693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:35:01.998555 6693 webserver.cc:460] Webserver started at http://127.28.104.194:46035/ using document root <none> and password file <none>
I20260619 06:35:01.999393 6693 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:01.999552 6693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:01.999923 6693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:35:02.003508 6693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "f2caef7e163c4be38b3ae00a22227664"
format_stamp: "Formatted at 2026-06-19 06:35:01 on dist-test-slave-nf4n"
server_key: "09310d101bad62e481f9e45e379dfab7"
server_key_iv: "469a1caff2973fd9de876c3a90885e40"
server_key_version: "encryptionkey@0"
I20260619 06:35:02.004431 6693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "f2caef7e163c4be38b3ae00a22227664"
format_stamp: "Formatted at 2026-06-19 06:35:01 on dist-test-slave-nf4n"
server_key: "09310d101bad62e481f9e45e379dfab7"
server_key_iv: "469a1caff2973fd9de876c3a90885e40"
server_key_version: "encryptionkey@0"
I20260619 06:35:02.010079 6693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 06:35:02.013911 6707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:02.015323 6693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:35:02.015509 6693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "f2caef7e163c4be38b3ae00a22227664"
format_stamp: "Formatted at 2026-06-19 06:35:01 on dist-test-slave-nf4n"
server_key: "09310d101bad62e481f9e45e379dfab7"
server_key_iv: "469a1caff2973fd9de876c3a90885e40"
server_key_version: "encryptionkey@0"
I20260619 06:35:02.015754 6693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:02.021682 6693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:02.022472 6693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:02.022755 6693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:02.023741 6693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:35:02.025427 6693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:35:02.025548 6693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:02.025662 6693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:35:02.025729 6693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:02.063942 6693 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:42639
I20260619 06:35:02.063995 6820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:42639 every 8 connection(s)
I20260619 06:35:02.065399 6693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:35:02.074025 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:02.068044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:57985 (local address 127.28.104.254:41483)
0619 06:35:02.068330 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:02.068334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:02.069893 (+ 1559us) server_negotiation.cc:407] Connection header received
0619 06:35:02.071624 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:02.071641 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:02.071711 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:02.071834 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:02.073209 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:02.073215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:02.073220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:02.073241 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:02.073300 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:02.073313 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:02.073444 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:02.073679 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:35:02.073750 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260619 06:35:02.074805 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 6693
I20260619 06:35:02.074975 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:35:02.075191 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:02.068703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:41483 (local address 127.28.104.194:57985)
0619 06:35:02.069739 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:02.069779 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:02.071345 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:02.072068 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:02.072078 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:02.072174 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:35:02.072824 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:35:02.072829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:35:02.072877 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:35:02.072982 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:35:02.073415 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:35:02.073426 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:35:02.073433 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:35:02.073724 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 06:35:02.073985 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":326,"threads_started":1}
I20260619 06:35:02.075590 29091 external_mini_cluster.cc:1428] Setting key 231b273a318748ceabd3ce741db7d09d
I20260619 06:35:02.076892 6821 heartbeater.cc:347] Connected to a master server at 127.28.104.254:41483
I20260619 06:35:02.077230 6821 heartbeater.cc:464] Registering TS with master...
I20260619 06:35:02.077939 6821 heartbeater.cc:511] Master 127.28.104.254:41483 requested a full tablet report, sending...
I20260619 06:35:02.078691 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:41483
--builtin_ntp_servers=127.28.104.212:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 06:35:02.079344 6501 ts_manager.cc:194] Registered new tserver with Master: f2caef7e163c4be38b3ae00a22227664 (127.28.104.194:42639)
W20260619 06:35:02.408689 6824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:02.408983 6824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:02.409150 6824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:02.416481 6824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:02.416591 6824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:02.416707 6824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:35:02.425940 6824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:41483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:02.427182 6824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:02.428874 6824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:02.441701 6829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:02.441843 6830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:02.441727 6832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:02.443358 6824 server_base.cc:1034] running on GCE node
I20260619 06:35:02.443933 6824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:02.444837 6824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:02.446035 6824 hybrid_clock.cc:648] HybridClock initialized: now 1781850902445970 us; error 60 us; skew 500 ppm
I20260619 06:35:02.446368 6824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:35:02.448686 6824 webserver.cc:460] Webserver started at http://127.28.104.195:40237/ using document root <none> and password file <none>
I20260619 06:35:02.449460 6824 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:02.449570 6824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:02.449901 6824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:35:02.452775 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "26bcaf5e7e6d4235830ae9724a455ad2"
format_stamp: "Formatted at 2026-06-19 06:35:02 on dist-test-slave-nf4n"
server_key: "3d2f8f806e3e5d1f3ab16a75a50bdbef"
server_key_iv: "90627942dcfaa09202ddb62ee07f4797"
server_key_version: "encryptionkey@0"
I20260619 06:35:02.453635 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "26bcaf5e7e6d4235830ae9724a455ad2"
format_stamp: "Formatted at 2026-06-19 06:35:02 on dist-test-slave-nf4n"
server_key: "3d2f8f806e3e5d1f3ab16a75a50bdbef"
server_key_iv: "90627942dcfaa09202ddb62ee07f4797"
server_key_version: "encryptionkey@0"
I20260619 06:35:02.458843 6824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:35:02.463135 6838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:02.464571 6824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:35:02.464779 6824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "26bcaf5e7e6d4235830ae9724a455ad2"
format_stamp: "Formatted at 2026-06-19 06:35:02 on dist-test-slave-nf4n"
server_key: "3d2f8f806e3e5d1f3ab16a75a50bdbef"
server_key_iv: "90627942dcfaa09202ddb62ee07f4797"
server_key_version: "encryptionkey@0"
I20260619 06:35:02.465029 6824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:02.471414 6824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:02.472267 6824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:02.472553 6824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:02.473667 6824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:35:02.475181 6824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:35:02.475298 6824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:02.475436 6824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:35:02.475533 6824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:02.510257 6824 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:37185
I20260619 06:35:02.510360 6951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:37185 every 8 connection(s)
I20260619 06:35:02.511799 6824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:35:02.521601 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:02.514754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:58247 (local address 127.28.104.254:41483)
0619 06:35:02.514946 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:02.514951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:02.516530 (+ 1579us) server_negotiation.cc:407] Connection header received
0619 06:35:02.518695 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:02.518702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:02.518794 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:02.518933 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:02.520571 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:02.520580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:02.520586 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:02.520611 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:02.520664 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:02.520681 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:02.520828 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:02.521167 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 06:35:02.521251 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:35:02.522329 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 6824
I20260619 06:35:02.522526 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:35:02.522994 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:02.515269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:41483 (local address 127.28.104.195:58247)
0619 06:35:02.516342 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:02.516391 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:02.518349 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:02.519203 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:02.519212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:02.519316 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:35:02.520211 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:35:02.520214 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:35:02.520259 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:35:02.520358 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:35:02.520806 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:35:02.520818 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:35:02.520856 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:35:02.521266 (+ 410us) client_negotiation.cc:240] Negotiation successful
0619 06:35:02.521575 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":321,"threads_started":1}
I20260619 06:35:02.523192 29091 external_mini_cluster.cc:1428] Setting key 1705a5aa44147735109b405f8f21f1c5
I20260619 06:35:02.524619 6952 heartbeater.cc:347] Connected to a master server at 127.28.104.254:41483
I20260619 06:35:02.524959 6952 heartbeater.cc:464] Registering TS with master...
I20260619 06:35:02.525734 6952 heartbeater.cc:511] Master 127.28.104.254:41483 requested a full tablet report, sending...
I20260619 06:35:02.527357 6501 ts_manager.cc:194] Registered new tserver with Master: 26bcaf5e7e6d4235830ae9724a455ad2 (127.28.104.195:37185)
I20260619 06:35:02.539544 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:35:02.545778 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:35:02.544137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.28.104.254:41483)
0619 06:35:02.544344 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:02.544352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:02.544832 (+ 480us) server_negotiation.cc:407] Connection header received
0619 06:35:02.544981 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:02.544984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:02.545096 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:02.545193 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:02.545511 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56272: BlockingRecv error: recv got EOF from 127.0.0.1:56272 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:35:02.546618 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 6561
I20260619 06:35:02.550591 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:02.547616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.28.104.193:36259)
0619 06:35:02.548178 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:02.548185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:02.548212 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:35:02.548271 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:02.548278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:02.548501 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:02.548680 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:02.549830 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:02.549837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:02.549850 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:02.549868 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:02.549908 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:02.549922 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:02.550097 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:02.550191 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 06:35:02.550300 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":255,"threads_started":1}
I20260619 06:35:02.551977 6684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:02.632278 6689 heartbeater.cc:503] Master 127.28.104.254:41483 was elected leader, sending a full tablet report...
I20260619 06:35:02.633616 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 6561
W20260619 06:35:02.649714 6485 connection.cc:535] server connection from 127.28.104.193:38737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:35:02.650028 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 6693
I20260619 06:35:02.653612 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:02.650937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45238 (local address 127.28.104.194:42639)
0619 06:35:02.651460 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:02.651467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:02.651494 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:35:02.651564 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:02.651570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:02.651793 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:02.651919 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:02.652784 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:02.652790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:02.652802 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:02.652814 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:02.652853 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:02.652871 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:02.652974 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:02.653144 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 06:35:02.653275 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":254,"threads_started":1}
I20260619 06:35:02.654805 6816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:02.734647 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 6693
I20260619 06:35:02.752388 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 6824
I20260619 06:35:02.756597 6965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:02.753371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.28.104.195:37185)
0619 06:35:02.754092 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:02.754099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:02.754137 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:35:02.754208 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:02.754215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:02.754450 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:02.754606 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:02.755845 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:02.755852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:02.755865 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:35:02.755879 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:02.755927 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:35:02.755959 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:02.756123 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:02.756180 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 06:35:02.756277 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":356,"threads_started":1}
I20260619 06:35:02.758081 6946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:02.843408 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 6824
I20260619 06:35:02.859469 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 6470
I20260619 06:35:02.860634 6532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:02.962877 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 6470
2026-06-19T06:35:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2427 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:35:03 dist-test-slave-nf4n krb5kdc[6970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:35:03 dist-test-slave-nf4n krb5kdc[6970](info): set up 2 sockets
Jun 19 06:35:03 dist-test-slave-nf4n krb5kdc[6970](info): commencing operation
krb5kdc: starting...
W20260619 06:35:04.069800 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:35:04 dist-test-slave-nf4n krb5kdc[6970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:35:04.300127 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:37575
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:37575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:35:04.621920 6983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:04.622203 6983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:04.622292 6983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:04.629272 6983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:35:04.629374 6983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:04.629417 6983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:04.629473 6983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:35:04.629516 6983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:35:04.639217 6983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:37575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:37575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:04.640491 6983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:04.641798 6983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:04.653240 6988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:04.653167 6989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:04.653311 6991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:04.654140 6983 server_base.cc:1034] running on GCE node
I20260619 06:35:04.655073 6983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:04.656455 6983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:04.657677 6983 hybrid_clock.cc:648] HybridClock initialized: now 1781850904657635 us; error 52 us; skew 500 ppm
I20260619 06:35:04.658006 6983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:35:04 dist-test-slave-nf4n krb5kdc[6970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850904, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:35:04.660913 6983 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:35:04.662412 6983 webserver.cc:460] Webserver started at http://127.28.104.254:42563/ using document root <none> and password file <none>
I20260619 06:35:04.663059 6983 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:04.663203 6983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:04.663496 6983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:35:04.666452 6983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "b1565e3c7b364d4991834466c24f80c8"
format_stamp: "Formatted at 2026-06-19 06:35:04 on dist-test-slave-nf4n"
server_key: "67dbaa2cd41911bcd30df2afb24a863b"
server_key_iv: "4c0b33a6919b6f55b300969091d790a6"
server_key_version: "encryptionkey@0"
I20260619 06:35:04.667203 6983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "b1565e3c7b364d4991834466c24f80c8"
format_stamp: "Formatted at 2026-06-19 06:35:04 on dist-test-slave-nf4n"
server_key: "67dbaa2cd41911bcd30df2afb24a863b"
server_key_iv: "4c0b33a6919b6f55b300969091d790a6"
server_key_version: "encryptionkey@0"
I20260619 06:35:04.672293 6983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:35:04.675634 6998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:04.676980 6983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:35:04.677194 6983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "b1565e3c7b364d4991834466c24f80c8"
format_stamp: "Formatted at 2026-06-19 06:35:04 on dist-test-slave-nf4n"
server_key: "67dbaa2cd41911bcd30df2afb24a863b"
server_key_iv: "4c0b33a6919b6f55b300969091d790a6"
server_key_version: "encryptionkey@0"
I20260619 06:35:04.677380 6983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:04.697690 6983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:04.701388 6983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:04.701666 6983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:04.720237 6983 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:37575
I20260619 06:35:04.720256 7049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:37575 every 8 connection(s)
I20260619 06:35:04.721707 6983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:35:04.722143 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 6983
I20260619 06:35:04.722366 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:35:04.723233 29091 external_mini_cluster.cc:1428] Setting key 4df18006fe333b96f927d8859860ac11
I20260619 06:35:04.726052 7050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:35:04.736085 7050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8: Bootstrap starting.
Jun 19 06:35:04 dist-test-slave-nf4n krb5kdc[6970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:35:04.739419 7050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8: Neither blocks nor log segments found. Creating new log.
I20260619 06:35:04.740480 7050 log.cc:824] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:35:04.743189 7050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8: No bootstrap required, opened a new log
I20260619 06:35:04.744825 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:04.725276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.28.104.254:37575)
0619 06:35:04.726000 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:04.726012 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:04.726055 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:35:04.727714 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:04.727735 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:04.728097 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:04.728500 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:04.729926 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:04.731324 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:04.732601 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:04.733341 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:04.733591 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:04.737565 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:04.737579 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:04.737593 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:04.737629 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:04.740860 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:04.741603 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:04.741610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:04.741615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:04.741775 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:04.742172 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:04.742177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:04.742179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:04.742715 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:04.743032 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:04.743471 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 06:35:04.743893 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":328,"threads_started":1}
I20260619 06:35:04.751394 7050 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1565e3c7b364d4991834466c24f80c8" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37575 } }
I20260619 06:35:04.751829 7050 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:35:04.751971 7050 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1565e3c7b364d4991834466c24f80c8, State: Initialized, Role: FOLLOWER
I20260619 06:35:04.752497 7050 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1565e3c7b364d4991834466c24f80c8" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37575 } }
I20260619 06:35:04.752694 7050 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:35:04.752789 7050 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:35:04.752908 7050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:35:04.755344 7050 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1565e3c7b364d4991834466c24f80c8" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37575 } }
I20260619 06:35:04.755775 7050 leader_election.cc:302] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1565e3c7b364d4991834466c24f80c8; no voters:
I20260619 06:35:04.756295 7050 leader_election.cc:288] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:35:04.756476 7055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:35:04.757221 7055 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [term 1 LEADER]: Becoming Leader. State: Replica: b1565e3c7b364d4991834466c24f80c8, State: Running, Role: LEADER
I20260619 06:35:04.757741 7055 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1565e3c7b364d4991834466c24f80c8" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37575 } }
I20260619 06:35:04.758303 7050 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:35:04.760365 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1565e3c7b364d4991834466c24f80c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1565e3c7b364d4991834466c24f80c8" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37575 } } }
I20260619 06:35:04.760798 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:35:04.760469 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1565e3c7b364d4991834466c24f80c8. Latest consensus state: current_term: 1 leader_uuid: "b1565e3c7b364d4991834466c24f80c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1565e3c7b364d4991834466c24f80c8" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37575 } } }
I20260619 06:35:04.760974 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:35:04.765689 7062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:35:04.771886 7062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:35:04.782270 7062 catalog_manager.cc:1347] Generated new cluster ID: 44ff4c52eaa94725858943a8675216bf
I20260619 06:35:04.782419 7062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:35:04.806195 7062 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:35:04.808112 7062 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:35:04.840169 7062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8: Generated new TSK 0
I20260619 06:35:04.841310 7062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:35:04.904736 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37575
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:35:05.229843 7076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:05.230118 7076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:05.230211 7076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:05.237265 7076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:05.237360 7076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:05.237479 7076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:35:05.247108 7076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:05.248425 7076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:05.249763 7076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:05.262653 7081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:05.262683 7082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:05.262848 7084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:05.264494 7076 server_base.cc:1034] running on GCE node
I20260619 06:35:05.265133 7076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:05.265954 7076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:05.267153 7076 hybrid_clock.cc:648] HybridClock initialized: now 1781850905267116 us; error 31 us; skew 500 ppm
I20260619 06:35:05.267484 7076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:35:05 dist-test-slave-nf4n krb5kdc[6970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850905, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:35:05.270613 7076 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:35:05.271983 7076 webserver.cc:460] Webserver started at http://127.28.104.193:36299/ using document root <none> and password file <none>
I20260619 06:35:05.272701 7076 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:05.272835 7076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:05.273169 7076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:35:05.275988 7076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "21bb007ecf694784ab4d725db8219092"
format_stamp: "Formatted at 2026-06-19 06:35:05 on dist-test-slave-nf4n"
server_key: "6cca9bb43eaabd4bdb7b5b08767d2b25"
server_key_iv: "9a96ba8a0d27f1f79182e571c0886415"
server_key_version: "encryptionkey@0"
I20260619 06:35:05.276679 7076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "21bb007ecf694784ab4d725db8219092"
format_stamp: "Formatted at 2026-06-19 06:35:05 on dist-test-slave-nf4n"
server_key: "6cca9bb43eaabd4bdb7b5b08767d2b25"
server_key_iv: "9a96ba8a0d27f1f79182e571c0886415"
server_key_version: "encryptionkey@0"
I20260619 06:35:05.281646 7076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:35:05.284906 7091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:05.286201 7076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:35:05.286352 7076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "21bb007ecf694784ab4d725db8219092"
format_stamp: "Formatted at 2026-06-19 06:35:05 on dist-test-slave-nf4n"
server_key: "6cca9bb43eaabd4bdb7b5b08767d2b25"
server_key_iv: "9a96ba8a0d27f1f79182e571c0886415"
server_key_version: "encryptionkey@0"
I20260619 06:35:05.286516 7076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:05.315464 7076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:05.319340 7076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:05.319645 7076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:05.320623 7076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:35:05.321972 7076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:35:05.322044 7076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:05.322208 7076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:35:05.322279 7076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:05.355085 7076 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:34375
I20260619 06:35:05.355130 7203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:34375 every 8 connection(s)
I20260619 06:35:05.356509 7076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:35:05.366927 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 7076
I20260619 06:35:05.367139 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:35:05.367985 29091 external_mini_cluster.cc:1428] Setting key 46e0b19e14809761f15171225c57010f
Jun 19 06:35:05 dist-test-slave-nf4n krb5kdc[6970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850905, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:35:05.378191 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:05.359045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:47497 (local address 127.28.104.254:37575)
0619 06:35:05.359527 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:05.359531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:05.360414 (+ 883us) server_negotiation.cc:407] Connection header received
0619 06:35:05.362270 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:05.362275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:05.362365 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:05.362483 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:05.363927 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:05.364662 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:05.366329 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:05.366737 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:05.366879 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:05.370883 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:05.370892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:05.370899 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:05.370941 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:05.374510 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:05.375371 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:05.375379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:05.375385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:05.375533 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:05.376020 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:05.376027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:05.376033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:05.376399 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:05.376555 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:05.377457 (+ 902us) server_negotiation.cc:299] Negotiation successful
0619 06:35:05.377730 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":261,"threads_started":1}
I20260619 06:35:05.379571 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:05.359608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37575 (local address 127.28.104.193:47497)
0619 06:35:05.360268 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:05.360305 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:05.361998 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:05.362722 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:05.362730 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:05.363214 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:35:05.363762 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:05.363780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:05.364805 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:05.364818 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:05.366165 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:05.366177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:05.366963 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:05.366971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:05.367260 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:05.368022 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:35:05.368044 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:35:05.370615 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:35:05.374690 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:05.374698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:05.374713 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:05.375221 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:05.375655 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:05.375658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:05.375661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:05.375881 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:05.376562 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:35:05.376569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:35:05.376946 (+ 377us) client_negotiation.cc:769] Sending connection context
0619 06:35:05.377312 (+ 366us) client_negotiation.cc:240] Negotiation successful
0619 06:35:05.377815 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":251,"threads_started":1}
I20260619 06:35:05.381356 7204 heartbeater.cc:347] Connected to a master server at 127.28.104.254:37575
I20260619 06:35:05.381668 7204 heartbeater.cc:464] Registering TS with master...
I20260619 06:35:05.382436 7204 heartbeater.cc:511] Master 127.28.104.254:37575 requested a full tablet report, sending...
I20260619 06:35:05.384588 7015 ts_manager.cc:194] Registered new tserver with Master: 21bb007ecf694784ab4d725db8219092 (127.28.104.193:34375)
I20260619 06:35:05.386087 7015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:47497
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:35:05.420310 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37575
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:35:05.740913 7210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:05.741257 7210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:05.741439 7210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:05.748582 7210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:05.748739 7210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:05.748929 7210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:35:05.758636 7210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:05.759959 7210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:05.761276 7210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:05.774204 7215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:05.774246 7216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:05.774278 7218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:05.775388 7210 server_base.cc:1034] running on GCE node
I20260619 06:35:05.776043 7210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:05.776990 7210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:05.778263 7210 hybrid_clock.cc:648] HybridClock initialized: now 1781850905778199 us; error 54 us; skew 500 ppm
I20260619 06:35:05.778605 7210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:35:05 dist-test-slave-nf4n krb5kdc[6970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850905, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:35:05.786218 7210 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:35:05.787734 7210 webserver.cc:460] Webserver started at http://127.28.104.194:35971/ using document root <none> and password file <none>
I20260619 06:35:05.788422 7210 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:05.788569 7210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:05.788880 7210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:35:05.792057 7210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "e6351e70f1d54866bb2e1708b868c768"
format_stamp: "Formatted at 2026-06-19 06:35:05 on dist-test-slave-nf4n"
server_key: "5fde1c6a17c977b7550005d4a97b4b90"
server_key_iv: "721fb608d65393a9a5b5a6744765e9e2"
server_key_version: "encryptionkey@0"
I20260619 06:35:05.792845 7210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "e6351e70f1d54866bb2e1708b868c768"
format_stamp: "Formatted at 2026-06-19 06:35:05 on dist-test-slave-nf4n"
server_key: "5fde1c6a17c977b7550005d4a97b4b90"
server_key_iv: "721fb608d65393a9a5b5a6744765e9e2"
server_key_version: "encryptionkey@0"
I20260619 06:35:05.797947 7210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:35:05.801518 7225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:05.802901 7210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:35:05.803081 7210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "e6351e70f1d54866bb2e1708b868c768"
format_stamp: "Formatted at 2026-06-19 06:35:05 on dist-test-slave-nf4n"
server_key: "5fde1c6a17c977b7550005d4a97b4b90"
server_key_iv: "721fb608d65393a9a5b5a6744765e9e2"
server_key_version: "encryptionkey@0"
I20260619 06:35:05.803284 7210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:05.830631 7210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:05.834524 7210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:05.834829 7210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:05.835737 7210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:35:05.837062 7210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:35:05.837203 7210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:05.837349 7210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:35:05.837421 7210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:05.870276 7210 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:42711
I20260619 06:35:05.870363 7338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:42711 every 8 connection(s)
I20260619 06:35:05.871631 7210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:35:05.881598 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 7210
I20260619 06:35:05.881776 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:35:05.882311 29091 external_mini_cluster.cc:1428] Setting key 75f436403de35d9d7f2a2ffe835161ba
Jun 19 06:35:05 dist-test-slave-nf4n krb5kdc[6970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850905, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:35:05.893566 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:05.874198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:41623 (local address 127.28.104.254:37575)
0619 06:35:05.874375 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:05.874380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:05.875479 (+ 1099us) server_negotiation.cc:407] Connection header received
0619 06:35:05.877295 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:05.877301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:05.877388 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:05.877498 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:05.878975 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:05.879681 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:05.881550 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:05.882237 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:05.882386 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:05.886195 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:05.886205 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:05.886214 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:05.886262 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:05.889795 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:05.890764 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:05.890772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:05.890778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:05.890994 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:05.891496 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:05.891503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:05.891508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:05.891837 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:05.891974 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:05.892952 (+ 978us) server_negotiation.cc:299] Negotiation successful
0619 06:35:05.893215 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:35:05.894547 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:05.874677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37575 (local address 127.28.104.194:41623)
0619 06:35:05.875331 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:05.875367 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:05.876972 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:05.877729 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:05.877737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:05.878206 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:35:05.878782 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:05.878798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:05.879821 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:05.879828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:05.881387 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:05.881403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:05.882476 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:05.882484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:05.882698 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:05.883476 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:35:05.883504 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:35:05.885962 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:35:05.889969 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:05.889981 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:05.890001 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:05.890629 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:05.891124 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:05.891132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:05.891138 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:05.891382 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:05.891967 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:35:05.891978 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:35:05.892456 (+ 478us) client_negotiation.cc:769] Sending connection context
0619 06:35:05.892774 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 06:35:05.893342 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":249,"threads_started":1}
I20260619 06:35:05.895956 7339 heartbeater.cc:347] Connected to a master server at 127.28.104.254:37575
I20260619 06:35:05.896284 7339 heartbeater.cc:464] Registering TS with master...
I20260619 06:35:05.897158 7339 heartbeater.cc:511] Master 127.28.104.254:37575 requested a full tablet report, sending...
I20260619 06:35:05.898715 7015 ts_manager.cc:194] Registered new tserver with Master: e6351e70f1d54866bb2e1708b868c768 (127.28.104.194:42711)
I20260619 06:35:05.899945 7015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:41623
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:35:05.944017 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37575
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:35:06.277307 7344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:06.277607 7344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:06.277819 7344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:06.285353 7344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:06.285462 7344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:06.285703 7344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:35:06.295899 7344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:06.297330 7344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:06.298687 7344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:06.312371 7351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:06.312589 7350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:06.312340 7353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:06.313138 7344 server_base.cc:1034] running on GCE node
I20260619 06:35:06.313740 7344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:06.314733 7344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:06.315956 7344 hybrid_clock.cc:648] HybridClock initialized: now 1781850906315911 us; error 63 us; skew 500 ppm
I20260619 06:35:06.316272 7344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:35:06 dist-test-slave-nf4n krb5kdc[6970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:35:06.319389 7344 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:35:06.320849 7344 webserver.cc:460] Webserver started at http://127.28.104.195:37047/ using document root <none> and password file <none>
I20260619 06:35:06.321581 7344 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:06.321733 7344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:06.322024 7344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:35:06.324945 7344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "98c40484ea0d47a59d7d4acdc85f05aa"
format_stamp: "Formatted at 2026-06-19 06:35:06 on dist-test-slave-nf4n"
server_key: "bc4c8770d5dabf757add962ae1b5c15d"
server_key_iv: "ecc9f215870a54479bb47ec7988634c9"
server_key_version: "encryptionkey@0"
I20260619 06:35:06.325783 7344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "98c40484ea0d47a59d7d4acdc85f05aa"
format_stamp: "Formatted at 2026-06-19 06:35:06 on dist-test-slave-nf4n"
server_key: "bc4c8770d5dabf757add962ae1b5c15d"
server_key_iv: "ecc9f215870a54479bb47ec7988634c9"
server_key_version: "encryptionkey@0"
I20260619 06:35:06.330754 7344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:35:06.334154 7360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:06.335454 7344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:35:06.335626 7344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "98c40484ea0d47a59d7d4acdc85f05aa"
format_stamp: "Formatted at 2026-06-19 06:35:06 on dist-test-slave-nf4n"
server_key: "bc4c8770d5dabf757add962ae1b5c15d"
server_key_iv: "ecc9f215870a54479bb47ec7988634c9"
server_key_version: "encryptionkey@0"
I20260619 06:35:06.335835 7344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:06.355154 7344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:06.359606 7344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:06.359932 7344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:06.360898 7344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:35:06.362228 7344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:35:06.362329 7344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:06.362476 7344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:35:06.362547 7344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:06.388926 7204 heartbeater.cc:503] Master 127.28.104.254:37575 was elected leader, sending a full tablet report...
I20260619 06:35:06.396908 7344 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:36021
I20260619 06:35:06.396957 7473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:36021 every 8 connection(s)
I20260619 06:35:06.398173 7344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:35:06.400671 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 7344
I20260619 06:35:06.400866 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:35:06.401381 29091 external_mini_cluster.cc:1428] Setting key 9666ad5afff0955f50f7bc00cb9feb77
Jun 19 06:35:06 dist-test-slave-nf4n krb5kdc[6970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:35:06.418574 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.400648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:35201 (local address 127.28.104.254:37575)
0619 06:35:06.401174 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:06.401180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:06.402289 (+ 1109us) server_negotiation.cc:407] Connection header received
0619 06:35:06.404479 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:06.404485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:06.404574 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:06.404696 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:06.406248 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.407088 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.408675 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.409133 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.409266 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.412614 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:06.412619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:06.412623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:06.412654 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:06.415437 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:06.416273 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:06.416279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:06.416282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:06.416453 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:06.416971 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:06.416976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:06.416979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:06.417343 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:06.417471 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:06.418130 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 06:35:06.418299 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":249,"threads_started":1}
I20260619 06:35:06.419500 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.401285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37575 (local address 127.28.104.195:35201)
0619 06:35:06.402099 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:06.402150 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:06.404157 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:06.404964 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:06.404972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:06.405511 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:35:06.406080 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.406098 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.407240 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.407250 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.408539 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.408551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.409366 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.409373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.409600 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.410279 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:35:06.410305 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:35:06.412385 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:35:06.415565 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:06.415576 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:06.415592 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:06.416110 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:06.416576 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:06.416581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:06.416584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:06.416837 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:06.417472 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:35:06.417478 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:35:06.417777 (+ 299us) client_negotiation.cc:769] Sending connection context
0619 06:35:06.418007 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 06:35:06.418364 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":354,"threads_started":1}
I20260619 06:35:06.420907 7474 heartbeater.cc:347] Connected to a master server at 127.28.104.254:37575
I20260619 06:35:06.421208 7474 heartbeater.cc:464] Registering TS with master...
I20260619 06:35:06.421908 7474 heartbeater.cc:511] Master 127.28.104.254:37575 requested a full tablet report, sending...
I20260619 06:35:06.423309 7015 ts_manager.cc:194] Registered new tserver with Master: 98c40484ea0d47a59d7d4acdc85f05aa (127.28.104.195:36021)
I20260619 06:35:06.424090 7015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:35201
I20260619 06:35:06.428966 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:35:06.445506 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.433249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.28.104.254:37575)
0619 06:35:06.433437 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:06.433441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:06.433879 (+ 438us) server_negotiation.cc:407] Connection header received
0619 06:35:06.434040 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:06.434044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:06.434108 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:06.434192 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:06.435476 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.436215 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.437466 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.437835 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.437928 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.440226 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:06.440232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:06.440235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:06.440269 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:06.443135 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:06.443727 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:06.443731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:06.443733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:06.443854 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:06.444185 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:06.444189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:06.444191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:06.444571 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:06.444778 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:06.445104 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 06:35:06.445285 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:35:06.449433 7015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:35:06.456146 7015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:35:06.482278 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.468069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.28.104.193:34375)
0619 06:35:06.469399 (+ 1330us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:06.469412 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:06.470470 (+ 1058us) server_negotiation.cc:407] Connection header received
0619 06:35:06.470825 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:06.470836 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:06.471167 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:06.471356 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:35:06.472552 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.474653 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.478311 (+ 3658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.480276 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.480487 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.480729 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:06.481561 (+ 832us) server_negotiation.cc:299] Negotiation successful
0619 06:35:06.481819 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":894,"threads_started":1}
I20260619 06:35:06.483619 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.467981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:34375 (local address 127.0.0.1:53246)
0619 06:35:06.470325 (+ 2344us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:06.470372 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:06.470577 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:06.471741 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:06.471752 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:06.471792 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:35:06.472287 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.472303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.474879 (+ 2576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.474887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.478089 (+ 3202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.478108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.480957 (+ 2849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.480966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.481241 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.481312 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 06:35:06.482871 (+ 1559us) client_negotiation.cc:240] Negotiation successful
0619 06:35:06.483145 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2077,"thread_start_us":1921,"threads_started":1}
I20260619 06:35:06.485559 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.468033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:36021 (local address 127.0.0.1:53320)
0619 06:35:06.470394 (+ 2361us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:06.470413 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:06.470570 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:06.472812 (+ 2242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:06.472820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:06.472854 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:35:06.473396 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.473411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.476389 (+ 2978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.476394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.480477 (+ 4083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.480537 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.483465 (+ 2928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.483474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.484960 (+ 1486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.484992 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:35:06.485092 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:35:06.485249 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2246,"thread_start_us":2099,"threads_started":1}
I20260619 06:35:06.486253 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.468725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.28.104.195:36021)
0619 06:35:06.469971 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:06.469980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:06.471864 (+ 1884us) server_negotiation.cc:407] Connection header received
0619 06:35:06.471992 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:06.472003 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:06.472281 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:06.472479 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:35:06.473620 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.476225 (+ 2605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.480735 (+ 4510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.482818 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.483068 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.483284 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:06.485860 (+ 2576us) server_negotiation.cc:299] Negotiation successful
0619 06:35:06.486001 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"thread_start_us":860,"threads_started":1}
I20260619 06:35:06.488243 7139 tablet_service.cc:1467] Processing CreateTablet for tablet 8410731bd20348b6992e622b6b12f2a7 (DEFAULT_TABLE table=test-table [id=b01f23c98ca74d3da630fcb1a48870c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:35:06.489681 7139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8410731bd20348b6992e622b6b12f2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:35:06.489620 7408 tablet_service.cc:1467] Processing CreateTablet for tablet 8410731bd20348b6992e622b6b12f2a7 (DEFAULT_TABLE table=test-table [id=b01f23c98ca74d3da630fcb1a48870c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:35:06.491102 7408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8410731bd20348b6992e622b6b12f2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:35:06.493561 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.470705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:42711 (local address 127.0.0.1:54690)
0619 06:35:06.484327 (+ 13622us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:06.484345 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:06.484453 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:06.485833 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:06.485838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:06.485863 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:35:06.486208 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.486222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.488310 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.488315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.490884 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.490899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.492717 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.492723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.492893 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.492936 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 06:35:06.493071 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 06:35:06.493208 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13480,"thread_start_us":13819,"threads_started":1}
I20260619 06:35:06.493608 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.470982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.28.104.194:42711)
0619 06:35:06.485224 (+ 14242us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:06.485232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:06.485262 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:35:06.485363 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:06.485374 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:06.485606 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:06.486414 (+ 808us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:35:06.486650 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.488136 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.491060 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.492489 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.492678 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.492841 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:06.493102 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 06:35:06.493298 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14070,"thread_start_us":13921,"threads_started":1}
I20260619 06:35:06.497531 7273 tablet_service.cc:1467] Processing CreateTablet for tablet 8410731bd20348b6992e622b6b12f2a7 (DEFAULT_TABLE table=test-table [id=b01f23c98ca74d3da630fcb1a48870c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:35:06.498950 7273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8410731bd20348b6992e622b6b12f2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:35:06.501952 7493 tablet_bootstrap.cc:492] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: Bootstrap starting.
I20260619 06:35:06.504976 7493 tablet_bootstrap.cc:654] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: Neither blocks nor log segments found. Creating new log.
I20260619 06:35:06.506645 7494 tablet_bootstrap.cc:492] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: Bootstrap starting.
I20260619 06:35:06.507716 7493 log.cc:824] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: Log is configured to *not* fsync() on all Append() calls
I20260619 06:35:06.510061 7494 tablet_bootstrap.cc:654] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: Neither blocks nor log segments found. Creating new log.
I20260619 06:35:06.510895 7493 tablet_bootstrap.cc:492] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: No bootstrap required, opened a new log
I20260619 06:35:06.511184 7493 ts_tablet_manager.cc:1397] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260619 06:35:06.511258 7494 log.cc:824] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: Log is configured to *not* fsync() on all Append() calls
I20260619 06:35:06.513417 7497 tablet_bootstrap.cc:492] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: Bootstrap starting.
I20260619 06:35:06.514173 7494 tablet_bootstrap.cc:492] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: No bootstrap required, opened a new log
I20260619 06:35:06.514420 7494 ts_tablet_manager.cc:1397] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 06:35:06.517217 7497 tablet_bootstrap.cc:654] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: Neither blocks nor log segments found. Creating new log.
I20260619 06:35:06.519810 7497 log.cc:824] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: Log is configured to *not* fsync() on all Append() calls
I20260619 06:35:06.522480 7493 raft_consensus.cc:348] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bb007ecf694784ab4d725db8219092" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 34375 } } peers { permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 } } peers { permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 } }
I20260619 06:35:06.523033 7493 raft_consensus.cc:374] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:35:06.523190 7493 raft_consensus.cc:729] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bb007ecf694784ab4d725db8219092, State: Initialized, Role: FOLLOWER
I20260619 06:35:06.523962 7493 consensus_queue.cc:260] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bb007ecf694784ab4d725db8219092" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 34375 } } peers { permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 } } peers { permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 } }
I20260619 06:35:06.524627 7494 raft_consensus.cc:348] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bb007ecf694784ab4d725db8219092" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 34375 } } peers { permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 } } peers { permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 } }
I20260619 06:35:06.525111 7494 raft_consensus.cc:374] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:35:06.525218 7494 raft_consensus.cc:729] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c40484ea0d47a59d7d4acdc85f05aa, State: Initialized, Role: FOLLOWER
I20260619 06:35:06.526268 7493 ts_tablet_manager.cc:1428] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260619 06:35:06.526378 7494 consensus_queue.cc:260] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bb007ecf694784ab4d725db8219092" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 34375 } } peers { permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 } } peers { permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 } }
I20260619 06:35:06.527736 7474 heartbeater.cc:503] Master 127.28.104.254:37575 was elected leader, sending a full tablet report...
I20260619 06:35:06.528497 7497 tablet_bootstrap.cc:492] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: No bootstrap required, opened a new log
I20260619 06:35:06.528776 7497 ts_tablet_manager.cc:1397] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260619 06:35:06.529588 7494 ts_tablet_manager.cc:1428] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 06:35:06.537096 7497 raft_consensus.cc:348] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bb007ecf694784ab4d725db8219092" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 34375 } } peers { permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 } } peers { permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 } }
I20260619 06:35:06.537423 7497 raft_consensus.cc:374] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:35:06.537492 7497 raft_consensus.cc:729] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6351e70f1d54866bb2e1708b868c768, State: Initialized, Role: FOLLOWER
I20260619 06:35:06.537935 7497 consensus_queue.cc:260] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bb007ecf694784ab4d725db8219092" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 34375 } } peers { permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 } } peers { permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 } }
I20260619 06:35:06.538913 7339 heartbeater.cc:503] Master 127.28.104.254:37575 was elected leader, sending a full tablet report...
I20260619 06:35:06.539433 7497 ts_tablet_manager.cc:1428] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
W20260619 06:35:06.608938 7205 tablet.cc:2369] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:35:06.623683 7340 tablet.cc:2369] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:35:06.644486 7499 raft_consensus.cc:482] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:35:06.644810 7499 raft_consensus.cc:504] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bb007ecf694784ab4d725db8219092" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 34375 } } peers { permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 } } peers { permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 } }
I20260619 06:35:06.646564 7499 leader_election.cc:288] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98c40484ea0d47a59d7d4acdc85f05aa (127.28.104.195:36021), e6351e70f1d54866bb2e1708b868c768 (127.28.104.194:42711)
W20260619 06:35:06.649786 7475 tablet.cc:2369] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:35:06.654533 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.647016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:56755 (local address 127.28.104.195:36021)
0619 06:35:06.647200 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:06.647204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:06.647898 (+ 694us) server_negotiation.cc:407] Connection header received
0619 06:35:06.647995 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:06.648000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:06.648081 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:06.648223 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:35:06.648848 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.649804 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.652386 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.653586 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.653754 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.653828 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:06.654169 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 06:35:06.654298 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:35:06.654639 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.647774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:49255 (local address 127.28.104.194:42711)
0619 06:35:06.648005 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:06.648011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:06.648298 (+ 287us) server_negotiation.cc:407] Connection header received
0619 06:35:06.648490 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:06.648495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:06.648573 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:06.648677 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:35:06.649283 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.650276 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.652570 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.653777 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.653911 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.653954 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:06.654292 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 06:35:06.654411 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:35:06.654735 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.647581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:42711 (local address 127.28.104.193:49255)
0619 06:35:06.648194 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:06.648219 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:06.648344 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:06.648709 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:06.648712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:06.648730 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:35:06.649137 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.649149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.650419 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.650426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.652443 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.652452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.653968 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.653974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.654154 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.654182 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:35:06.654256 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:35:06.654351 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":1,"thread_start_us":307,"threads_started":1}
I20260619 06:35:06.655045 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.646858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:36021 (local address 127.28.104.193:56755)
0619 06:35:06.647518 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:06.647537 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:06.647695 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:06.648238 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:06.648242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:06.648287 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:35:06.648696 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.648710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.650028 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.650031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.652231 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:06.652245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.653794 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.653798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:06.654002 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.654039 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:35:06.654139 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:35:06.654277 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":30,"thread_start_us":326,"threads_started":1}
I20260619 06:35:06.655768 7293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410731bd20348b6992e622b6b12f2a7" candidate_uuid: "21bb007ecf694784ab4d725db8219092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6351e70f1d54866bb2e1708b868c768" is_pre_election: true
I20260619 06:35:06.655961 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410731bd20348b6992e622b6b12f2a7" candidate_uuid: "21bb007ecf694784ab4d725db8219092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" is_pre_election: true
I20260619 06:35:06.656344 7293 raft_consensus.cc:2454] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21bb007ecf694784ab4d725db8219092 in term 0.
I20260619 06:35:06.656418 7429 raft_consensus.cc:2454] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21bb007ecf694784ab4d725db8219092 in term 0.
I20260619 06:35:06.657107 7092 leader_election.cc:302] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21bb007ecf694784ab4d725db8219092, e6351e70f1d54866bb2e1708b868c768; no voters:
I20260619 06:35:06.657467 7499 raft_consensus.cc:2781] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:35:06.657605 7499 raft_consensus.cc:482] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:35:06.657696 7499 raft_consensus.cc:3037] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:35:06.660219 7499 raft_consensus.cc:504] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bb007ecf694784ab4d725db8219092" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 34375 } } peers { permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 } } peers { permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 } }
I20260619 06:35:06.660765 7499 leader_election.cc:288] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [CANDIDATE]: Term 1 election: Requested vote from peers 98c40484ea0d47a59d7d4acdc85f05aa (127.28.104.195:36021), e6351e70f1d54866bb2e1708b868c768 (127.28.104.194:42711)
I20260619 06:35:06.661203 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410731bd20348b6992e622b6b12f2a7" candidate_uuid: "21bb007ecf694784ab4d725db8219092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c40484ea0d47a59d7d4acdc85f05aa"
I20260619 06:35:06.661266 7293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8410731bd20348b6992e622b6b12f2a7" candidate_uuid: "21bb007ecf694784ab4d725db8219092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6351e70f1d54866bb2e1708b868c768"
I20260619 06:35:06.661381 7429 raft_consensus.cc:3037] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:35:06.661505 7293 raft_consensus.cc:3037] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:35:06.663501 7429 raft_consensus.cc:2454] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21bb007ecf694784ab4d725db8219092 in term 1.
I20260619 06:35:06.663749 7293 raft_consensus.cc:2454] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21bb007ecf694784ab4d725db8219092 in term 1.
I20260619 06:35:06.663887 7092 leader_election.cc:302] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21bb007ecf694784ab4d725db8219092, 98c40484ea0d47a59d7d4acdc85f05aa; no voters:
I20260619 06:35:06.664161 7499 raft_consensus.cc:2781] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:35:06.664589 7499 raft_consensus.cc:686] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 1 LEADER]: Becoming Leader. State: Replica: 21bb007ecf694784ab4d725db8219092, State: Running, Role: LEADER
I20260619 06:35:06.665131 7499 consensus_queue.cc:237] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bb007ecf694784ab4d725db8219092" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 34375 } } peers { permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 } } peers { permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 } }
I20260619 06:35:06.669446 7013 catalog_manager.cc:5495] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21bb007ecf694784ab4d725db8219092 (127.28.104.193). New cstate: current_term: 1 leader_uuid: "21bb007ecf694784ab4d725db8219092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bb007ecf694784ab4d725db8219092" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 34375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 } health_report { overall_health: UNKNOWN } } }
I20260619 06:35:06.716194 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.711149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.28.104.193:34375)
0619 06:35:06.711416 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:06.711420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:06.711439 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:35:06.711668 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:06.711673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:06.711766 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:06.711890 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:35:06.712180 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.712918 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.714228 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.714656 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.714750 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.715137 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:35:06.715665 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:35:06.715774 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:06.715834 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 06:35:06.715900 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260619 06:35:06.724644 7429 raft_consensus.cc:1261] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa [term 1 FOLLOWER]: Refusing update from remote peer 21bb007ecf694784ab4d725db8219092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:35:06.724643 7293 raft_consensus.cc:1261] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768 [term 1 FOLLOWER]: Refusing update from remote peer 21bb007ecf694784ab4d725db8219092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:35:06.725706 7499 consensus_queue.cc:1035] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6351e70f1d54866bb2e1708b868c768" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:35:06.726027 7504 consensus_queue.cc:1035] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c40484ea0d47a59d7d4acdc85f05aa" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 36021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:35:06.742107 7507 mvcc.cc:205] Tried to move back new op lower bound from 7298461313932668928 to 7298461313703759872. Current Snapshot: MvccSnapshot[applied={T|T < 7298461313932668928 or (T in {7298461313932668928})}]
I20260619 06:35:06.755414 7013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:35:06.755716 7013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:35:06.759171 7013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1565e3c7b364d4991834466c24f80c8: Sending DeleteTablet for 3 replicas of tablet 8410731bd20348b6992e622b6b12f2a7
I20260619 06:35:06.760254 7139 tablet_service.cc:1514] Processing DeleteTablet for tablet 8410731bd20348b6992e622b6b12f2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:35:06 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:53246
I20260619 06:35:06.760442 7408 tablet_service.cc:1514] Processing DeleteTablet for tablet 8410731bd20348b6992e622b6b12f2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:35:06 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:53320
I20260619 06:35:06.760653 7273 tablet_service.cc:1514] Processing DeleteTablet for tablet 8410731bd20348b6992e622b6b12f2a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:35:06 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:54690
I20260619 06:35:06.763444 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 7076
I20260619 06:35:06.777561 7517 tablet_replica.cc:330] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: stopping tablet replica
I20260619 06:35:06.777951 7517 raft_consensus.cc:2229] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:35:06.778594 7517 raft_consensus.cc:2258] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:35:06.780558 7518 tablet_replica.cc:330] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: stopping tablet replica
I20260619 06:35:06.780582 7519 tablet_replica.cc:330] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: stopping tablet replica
I20260619 06:35:06.781008 7519 raft_consensus.cc:2229] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:35:06.781354 7518 raft_consensus.cc:2229] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:35:06.781553 7519 raft_consensus.cc:2258] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:35:06.781801 7518 raft_consensus.cc:2258] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:35:06.782704 7517 ts_tablet_manager.cc:1905] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:35:06.783896 7518 ts_tablet_manager.cc:1905] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:35:06.787216 7519 ts_tablet_manager.cc:1905] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:35:06.791023 7518 ts_tablet_manager.cc:1918] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:35:06.791244 7518 log.cc:1196] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/8410731bd20348b6992e622b6b12f2a7
I20260619 06:35:06.791749 7518 ts_tablet_manager.cc:1939] T 8410731bd20348b6992e622b6b12f2a7 P 98c40484ea0d47a59d7d4acdc85f05aa: Deleting consensus metadata
I20260619 06:35:06.793143 7517 ts_tablet_manager.cc:1918] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:35:06.793320 7517 log.cc:1196] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/8410731bd20348b6992e622b6b12f2a7
I20260619 06:35:06.793830 7517 ts_tablet_manager.cc:1939] T 8410731bd20348b6992e622b6b12f2a7 P 21bb007ecf694784ab4d725db8219092: Deleting consensus metadata
I20260619 06:35:06.793906 6999 catalog_manager.cc:4841] TS 98c40484ea0d47a59d7d4acdc85f05aa (127.28.104.195:36021): tablet 8410731bd20348b6992e622b6b12f2a7 (table test-table [id=b01f23c98ca74d3da630fcb1a48870c9]) successfully deleted
I20260619 06:35:06.794374 7519 ts_tablet_manager.cc:1918] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:35:06.794528 7519 log.cc:1196] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/8410731bd20348b6992e622b6b12f2a7
I20260619 06:35:06.795011 7519 ts_tablet_manager.cc:1939] T 8410731bd20348b6992e622b6b12f2a7 P e6351e70f1d54866bb2e1708b868c768: Deleting consensus metadata
I20260619 06:35:06.795357 7002 catalog_manager.cc:4841] TS 21bb007ecf694784ab4d725db8219092 (127.28.104.193:34375): tablet 8410731bd20348b6992e622b6b12f2a7 (table test-table [id=b01f23c98ca74d3da630fcb1a48870c9]) successfully deleted
I20260619 06:35:06.796716 6999 catalog_manager.cc:4841] TS e6351e70f1d54866bb2e1708b868c768 (127.28.104.194:42711): tablet 8410731bd20348b6992e622b6b12f2a7 (table test-table [id=b01f23c98ca74d3da630fcb1a48870c9]) successfully deleted
Jun 19 06:35:06 dist-test-slave-nf4n krb5kdc[6970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:35:06.806278 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.766201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53268 (local address 127.28.104.193:34375)
0619 06:35:06.766734 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:06.766743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:06.779902 (+ 13159us) server_negotiation.cc:407] Connection header received
0619 06:35:06.782203 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:06.782211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:06.782302 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:06.782364 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:06.784523 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.785703 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.788413 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.789209 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.789314 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.799099 (+ 9785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:06.799113 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:06.799133 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:06.799183 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:06.803192 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:06.803913 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:06.803924 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:06.803932 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:06.804126 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:06.804522 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:06.804527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:06.804531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:06.804990 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:06.805336 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:06.805611 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 06:35:06.805857 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
I20260619 06:35:06.807127 7199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:06.913756 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 7076
I20260619 06:35:06.932040 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 7210
Jun 19 06:35:06 dist-test-slave-nf4n krb5kdc[6970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:35:06.947273 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:06.932851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.28.104.194:42711)
0619 06:35:06.933059 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:06.933067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:06.933087 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:35:06.933339 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:06.933345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:06.933418 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:06.933527 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:06.934744 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.935499 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.936765 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:06.937226 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:06.937359 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:06.941571 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:06.941583 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:06.941598 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:06.941637 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:06.944302 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:06.944975 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:06.944992 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:06.944994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:06.945325 (+ 331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:06.945688 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:06.945696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:06.945700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:06.946099 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:06.946400 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:06.946712 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 06:35:06.946926 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:35:06.948021 7334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:07.036413 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 7210
I20260619 06:35:07.052879 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 7344
Jun 19 06:35:07 dist-test-slave-nf4n krb5kdc[6970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:35:07.071511 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:07.053776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.28.104.195:36021)
0619 06:35:07.053947 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:07.053956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:07.054015 (+ 59us) server_negotiation.cc:407] Connection header received
0619 06:35:07.054242 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:07.054246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:07.054317 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:07.054425 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:07.055652 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:07.056349 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:07.057744 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:07.058190 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:07.058330 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:07.066084 (+ 7754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:07.066100 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:07.066116 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:07.066150 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:07.068731 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:07.069476 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:07.069483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:07.069485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:07.069618 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:07.070003 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:07.070008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:07.070010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:07.070340 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:07.070633 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:07.070939 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 06:35:07.071196 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:35:07.072253 7469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:07.161365 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 7344
I20260619 06:35:07.178040 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 6983
I20260619 06:35:07.179172 7045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:07.288643 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 6983
2026-06-19T06:35:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4325 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:35:07 dist-test-slave-nf4n krb5kdc[7527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:35:07 dist-test-slave-nf4n krb5kdc[7527](info): set up 2 sockets
Jun 19 06:35:07 dist-test-slave-nf4n krb5kdc[7527](info): commencing operation
krb5kdc: starting...
W20260619 06:35:08.393851 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:35:08 dist-test-slave-nf4n krb5kdc[7527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:35:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.104.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.104.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/rangeradmin_127.28.104.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/rangeradmin_127.28.104.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.104.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.104.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/rangerlookup_127.28.104.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/rangerlookup_127.28.104.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.104.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.104.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/HTTP_127.28.104.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/HTTP_127.28.104.212@KRBTEST.COM.keytab.
I20260619 06:35:08.722195 29091 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskS2JAv5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/postgres -l logfile start
2026-06-19 06:35:11.163 UTC [7557] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:35:11.163 UTC [7557] LOG: listening on IPv4 address "127.28.104.212", port 53031
2026-06-19 06:35:11.166 UTC [7557] LOG: listening on Unix socket "/tmp/.s.PGSQL.53031"
2026-06-19 06:35:11.189 UTC [7560] LOG: database system was shut down at 2026-06-19 06:35:09 UTC
2026-06-19 06:35:11.193 UTC [7557] LOG: database system is ready to accept connections
I20260619 06:35:12.177165 29091 mini_postgres.cc:96] Postgres bound to 53031
2026-06-19 06:35:12.195 UTC [7568] FATAL: database "slave" does not exist
127.28.104.212:53031 - accepting connections
I20260619 06:35:12.195925 29091 mini_ranger.cc:161] Starting Ranger...
I20260619 06:35:12.222404 29091 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:35:12.680967 29091 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:35:12.681160 29091 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-admin
2026-06-19 06:35:12,981 [I] DB FLAVOR :POSTGRES
2026-06-19 06:35:12,982 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:35:12,982 [I] Checking connection..
2026-06-19 06:35:12,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:35:13,220 [I] Checking connection passed.
2026-06-19 06:35:13,220 [I] --------- Verifying version history table ---------
2026-06-19 06:35:13,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:35:13,471 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:35:13,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:35:13,715 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:35:13,715 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:35:13,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:35:13.952 UTC [7632] WARNING: there is no transaction in progress
2026-06-19 06:35:13,958 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:35:13,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:35:14,241 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:35:14,242 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:35:14,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:35:14,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:35:14,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nf4n.c.gcp-upstream.internal','N') ;"
2026-06-19 06:35:14,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:35:15,210 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:35:15,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:35:15,471 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:35:15,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:35:15,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:35:15,956 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:35:15,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:35:16.908 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:16.924 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:16.940 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:16.955 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:16.970 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:16.988 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17.043 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17.049 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17.063 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17.072 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17.082 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17.095 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17.104 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17.111 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17.117 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17.123 UTC [7741] WARNING: there is no transaction in progress
2026-06-19 06:35:17,723 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:35:17,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:35:17,961 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:35:17,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:35:18,194 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:35:18,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:35:18,434 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:35:18,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:35:18,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:35:18,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nf4n.c.gcp-upstream.internal';"
2026-06-19 06:35:19,156 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:35:19,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:35:19,431 [I] DB_PATCHES have already been applied
I20260619 06:35:19.436861 29091 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-admin:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskS2JAv5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:35:19.437012 29091 mini_ranger.cc:193] Using host: 127.28.104.212
I20260619 06:35:19.451637 29091 mini_ranger.cc:227] Ranger admin URL: http://127.28.104.212:54565
Jun 19, 2026 6:35:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:35:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:35:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:35:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:35:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:35:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:35:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:35:20.605643 29091 mini_ranger.cc:160] Time spent starting Ranger: real 8.410s user 0.003s sys 0.036s
Jun 19, 2026 6:35:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:35:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 6:35:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:35:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:35:40 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:35:41.114002 29091 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.104.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.104.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/rangerkms_127.28.104.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/rangerkms_127.28.104.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.104.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.104.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.104.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/HTTP_127.28.104.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.104.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/HTTP_127.28.104.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:35:41 dist-test-slave-nf4n krb5kdc[7527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850941, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:35:41.288926 29091 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:35:41.312724 29091 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:35:42.166623 29091 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:35:42.166772 29091 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-kms
I20260619 06:35:42.166937 29091 mini_ranger_kms.cc:227] Using postgres at 127.28.104.212:53031
2026-06-19 06:35:42,590 [I] DB FLAVOR :POSTGRES
2026-06-19 06:35:42,590 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:35:42,590 [I] Checking connection
2026-06-19 06:35:42,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:35:42,828 [I] connection success
2026-06-19 06:35:42,828 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:35:42,828 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:35:42,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:35:43,069 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:35:43,070 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:35:43,070 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:35:43,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:53031/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:35:43,323 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:35:43.791780 29091 mini_ranger_kms.cc:310] Created kms service
I20260619 06:35:43.993764 29091 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:35:44.037587 29091 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:35:44 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850941, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:35:44.425401 29091 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:35:44.425746 29091 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-kms:/tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:35:44.425814 29091 mini_ranger_kms.cc:240] Using host: 127.28.104.212
I20260619 06:35:44.447441 29091 mini_ranger_kms.cc:276] Ranger KMS PID: 7922
I20260619 06:35:44.447762 29091 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.104.212:41177
Jun 19, 2026 6:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 06:35:46 dist-test-slave-nf4n krb5kdc[7527](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781850946, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.104.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:35:46.746737 29091 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.458s user 0.159s sys 0.285s
I20260619 06:35:46.746975 29091 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:35:46.747180 29091 mini_ranger_kms.cc:401] 127.28.104.212:41177/kms/v1/keys
Jun 19 06:35:47 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850946, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.104.212@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
Jun 19 06:35:47 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850946, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.104.212@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
Jun 19 06:35:49 dist-test-slave-nf4n krb5kdc[7527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:35:49.845602 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:34397
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:34397
--ranger_config_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:41177/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:35:50.179220 7974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:50.179574 7974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:50.179718 7974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:50.187064 7974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:35:50.187204 7974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:50.187300 7974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:50.187404 7974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:35:50.187511 7974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:35:50.197711 7974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.104.212:41177/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:34397
--ranger_config_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:34397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:50.199031 7974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:50.200345 7974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:50.213706 7980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:50.214087 7982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:50.214140 7979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:50.214833 7974 server_base.cc:1034] running on GCE node
I20260619 06:35:50.215844 7974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:50.217793 7974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:50.219054 7974 hybrid_clock.cc:648] HybridClock initialized: now 1781850950218991 us; error 60 us; skew 500 ppm
I20260619 06:35:50.219457 7974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:35:50 dist-test-slave-nf4n krb5kdc[7527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850950, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:35:50.226029 7974 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:35:50.227624 7974 webserver.cc:460] Webserver started at http://127.28.104.254:37487/ using document root <none> and password file <none>
I20260619 06:35:50.228279 7974 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:50.228371 7974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:50.228711 7974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:35:50 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850950, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:35:50.276017 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "0ce1e55ef3d34a4cbe3654229e2cc729"
format_stamp: "Formatted at 2026-06-19 06:35:50 on dist-test-slave-nf4n"
server_key: "c001d21317428d33c1ad4ae79a26e9da"
server_key_iv: "68f8a2693f7671a3afe47448bef34d26"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:50.276856 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "0ce1e55ef3d34a4cbe3654229e2cc729"
format_stamp: "Formatted at 2026-06-19 06:35:50 on dist-test-slave-nf4n"
server_key: "c001d21317428d33c1ad4ae79a26e9da"
server_key_iv: "68f8a2693f7671a3afe47448bef34d26"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:50.282239 7974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:35:50.302196 7989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:50.303771 7974 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:35:50.303982 7974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "0ce1e55ef3d34a4cbe3654229e2cc729"
format_stamp: "Formatted at 2026-06-19 06:35:50 on dist-test-slave-nf4n"
server_key: "c001d21317428d33c1ad4ae79a26e9da"
server_key_iv: "68f8a2693f7671a3afe47448bef34d26"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:50.304214 7974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:50.318234 7974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:50.322475 7974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:50.322796 7974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:50.343350 7974 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:34397
I20260619 06:35:50.343372 8040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:34397 every 8 connection(s)
I20260619 06:35:50.344866 7974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:35:50.345790 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 7974
I20260619 06:35:50.346029 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:35:50.347033 29091 external_mini_cluster.cc:1428] Setting key ea2bf8393d68a719eb8760cdb00cc3f0
I20260619 06:35:50.349692 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:35:50.364173 8041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729: Bootstrap starting.
I20260619 06:35:50.368307 8041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729: Neither blocks nor log segments found. Creating new log.
I20260619 06:35:50.369685 8041 log.cc:824] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729: Log is configured to *not* fsync() on all Append() calls
I20260619 06:35:50.373095 8041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729: No bootstrap required, opened a new log
Jun 19 06:35:50 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:35:50.383432 8041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1e55ef3d34a4cbe3654229e2cc729" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34397 } }
I20260619 06:35:50.383869 8041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:35:50.383973 8041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce1e55ef3d34a4cbe3654229e2cc729, State: Initialized, Role: FOLLOWER
I20260619 06:35:50.384482 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:50.354441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.28.104.254:34397)
0619 06:35:50.365212 (+ 10771us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:50.365225 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:50.365267 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:35:50.366997 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:50.367018 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:50.367404 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:50.367798 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:50.369349 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:50.371137 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:50.372375 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:50.373107 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:50.373855 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:50.377130 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:50.377143 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:50.377160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:50.377193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:50.380219 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:50.380914 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:50.380920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:50.380927 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:50.381148 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:50.381525 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:50.381532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:50.381535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:50.382088 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:50.382403 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:50.382835 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 06:35:50.383272 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10484,"thread_start_us":13700,"threads_started":1}
I20260619 06:35:50.384614 8041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1e55ef3d34a4cbe3654229e2cc729" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34397 } }
I20260619 06:35:50.384924 8041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:35:50.385114 8041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:35:50.385309 8041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:35:50.388501 8041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1e55ef3d34a4cbe3654229e2cc729" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34397 } }
I20260619 06:35:50.389043 8041 leader_election.cc:302] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ce1e55ef3d34a4cbe3654229e2cc729; no voters:
I20260619 06:35:50.389691 8041 leader_election.cc:288] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:35:50.389856 8046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:35:50.391676 8041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:35:50.397399 8046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce1e55ef3d34a4cbe3654229e2cc729, State: Running, Role: LEADER
I20260619 06:35:50.398149 8046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1e55ef3d34a4cbe3654229e2cc729" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34397 } }
I20260619 06:35:50.401841 8047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ce1e55ef3d34a4cbe3654229e2cc729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1e55ef3d34a4cbe3654229e2cc729" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34397 } } }
I20260619 06:35:50.402194 8047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [sys.catalog]: This master's current role is: LEADER
I20260619 06:35:50.402711 8048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ce1e55ef3d34a4cbe3654229e2cc729. Latest consensus state: current_term: 1 leader_uuid: "0ce1e55ef3d34a4cbe3654229e2cc729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1e55ef3d34a4cbe3654229e2cc729" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34397 } } }
I20260619 06:35:50.403039 8048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729 [sys.catalog]: This master's current role is: LEADER
I20260619 06:35:50.403249 8058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:35:50.418205 8058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:35:50.418985 8041 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:35:50.447846 8058 catalog_manager.cc:1347] Generated new cluster ID: 9cd9ae06290e4527b5c3fd0d009f9e74
I20260619 06:35:50.448045 8058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:35:50.475940 8058 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:35:50.478142 8058 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:35:50.491777 8058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729: Generated new TSK 0
I20260619 06:35:50.493008 8058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:35:51.282719 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:41177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34397
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:35:51 dist-test-slave-nf4n krb5kdc[7527](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781850951, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:35:51.685592 8086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:51.685973 8086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:51.686179 8086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:51.697721 8086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:51.697853 8086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:51.698084 8086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:35:51.713753 8086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.104.212:41177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:51.715481 8086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:51.717298 8086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:51.734338 8096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:51.735188 8097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:51.743184 8099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:51.743757 8086 server_base.cc:1034] running on GCE node
I20260619 06:35:51.744552 8086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:51.749593 8086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:51.750913 8086 hybrid_clock.cc:648] HybridClock initialized: now 1781850951750809 us; error 86 us; skew 500 ppm
I20260619 06:35:51.751346 8086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:35:51 dist-test-slave-nf4n krb5kdc[7527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850951, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:35:51.755883 8086 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:35:51.764366 8086 webserver.cc:460] Webserver started at http://127.28.104.193:33479/ using document root <none> and password file <none>
I20260619 06:35:51.765511 8086 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:51.765666 8086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:51.766027 8086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:35:51 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850951, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:35:51.798272 8086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "f38922b7333743218b064758c736c6a8"
format_stamp: "Formatted at 2026-06-19 06:35:51 on dist-test-slave-nf4n"
server_key: "4d38af7e7d4b421d775403c56446606d"
server_key_iv: "61e6dc1cf94767faf18127c266914a4b"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:51.799468 8086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "f38922b7333743218b064758c736c6a8"
format_stamp: "Formatted at 2026-06-19 06:35:51 on dist-test-slave-nf4n"
server_key: "4d38af7e7d4b421d775403c56446606d"
server_key_iv: "61e6dc1cf94767faf18127c266914a4b"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:51.807209 8086 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260619 06:35:51.833673 8106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:51.835705 8086 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.007s sys 0.000s
I20260619 06:35:51.835976 8086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "f38922b7333743218b064758c736c6a8"
format_stamp: "Formatted at 2026-06-19 06:35:51 on dist-test-slave-nf4n"
server_key: "4d38af7e7d4b421d775403c56446606d"
server_key_iv: "61e6dc1cf94767faf18127c266914a4b"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:51.836340 8086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:51.870990 8086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:51.875653 8086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:51.876093 8086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:51.880968 8086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:35:51.882754 8086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:35:51.882858 8086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:51.882990 8086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:35:51.883088 8086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:51.924206 8086 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:35089
I20260619 06:35:51.924222 8220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:35089 every 8 connection(s)
I20260619 06:35:51.925679 8086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:35:51.932879 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 8086
I20260619 06:35:51.933106 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:35:51.933920 29091 external_mini_cluster.cc:1428] Setting key 67128554576168375d7e29ef4e6c4a47
Jun 19 06:35:51 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850951, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
I20260619 06:35:51.980362 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:51.945778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:59167 (local address 127.28.104.254:34397)
0619 06:35:51.958314 (+ 12536us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:51.958321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:51.958349 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:35:51.961150 (+ 2801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:51.961157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:51.961256 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:51.961393 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:51.963501 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:51.964700 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:51.967285 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:51.967949 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:51.968081 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:51.972757 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:51.972765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:51.972770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:51.972812 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:51.976448 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:51.977444 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:51.977449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:51.977453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:51.977613 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:51.978155 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:51.978159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:51.978162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:51.978531 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:51.978677 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:51.979728 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0619 06:35:51.979965 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12395,"thread_start_us":12269,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
I20260619 06:35:51.985612 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:51.946490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34397 (local address 127.28.104.193:59167)
0619 06:35:51.955337 (+ 8847us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:51.955399 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:51.960749 (+ 5350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:51.961699 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:51.961712 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:51.962344 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:35:51.963263 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:51.963292 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:51.964907 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:51.964913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:51.967075 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:51.967093 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:51.968395 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:51.968403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:51.968729 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:51.969567 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:35:51.969603 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:35:51.972471 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:35:51.976630 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:51.976640 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:51.976659 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:51.977301 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:51.977754 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:51.977759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:51.977762 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:51.978007 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:51.978774 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:35:51.978782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:35:51.979254 (+ 472us) client_negotiation.cc:769] Sending connection context
0619 06:35:51.979596 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 06:35:51.980180 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8481,"thread_start_us":10415,"threads_started":1}
I20260619 06:35:51.987709 8221 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34397
I20260619 06:35:51.988068 8221 heartbeater.cc:464] Registering TS with master...
I20260619 06:35:51.988977 8221 heartbeater.cc:511] Master 127.28.104.254:34397 requested a full tablet report, sending...
I20260619 06:35:51.991671 8006 ts_manager.cc:194] Registered new tserver with Master: f38922b7333743218b064758c736c6a8 (127.28.104.193:35089)
I20260619 06:35:51.994065 8006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:59167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:35:52.014427 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:41177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34397
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:35:52 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850951, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
W20260619 06:35:52.488377 8227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:52.488898 8227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:52.489142 8227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:52.500721 8227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:52.500929 8227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:52.501200 8227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
Jun 19 06:35:52 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850951, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:35:52.517925 8227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.104.212:41177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:52.519914 8227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:52.521768 8227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:52.540796 8234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:52.545209 8233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:52.545465 8227 server_base.cc:1034] running on GCE node
W20260619 06:35:52.545713 8236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:52.546463 8227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:52.547935 8227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:52.549197 8227 hybrid_clock.cc:648] HybridClock initialized: now 1781850952549123 us; error 56 us; skew 500 ppm
I20260619 06:35:52.549642 8227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:35:52 dist-test-slave-nf4n krb5kdc[7527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850952, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:35:52.553898 8227 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:35:52.556761 8227 webserver.cc:460] Webserver started at http://127.28.104.194:36591/ using document root <none> and password file <none>
I20260619 06:35:52.557710 8227 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:52.557847 8227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:52.558193 8227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:35:52 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850952, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:35:52.575974 8227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "4abed0a0edc345eb85aec2bbff01fe2e"
format_stamp: "Formatted at 2026-06-19 06:35:52 on dist-test-slave-nf4n"
server_key: "ca3111c302a8ccf60958bcfd002be684"
server_key_iv: "43cc156d9ca8fa2c714535df4bc573ee"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:52.577428 8227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "4abed0a0edc345eb85aec2bbff01fe2e"
format_stamp: "Formatted at 2026-06-19 06:35:52 on dist-test-slave-nf4n"
server_key: "ca3111c302a8ccf60958bcfd002be684"
server_key_iv: "43cc156d9ca8fa2c714535df4bc573ee"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:52.626988 8227 fs_manager.cc:691] Time spent creating directory manager: real 0.049s user 0.023s sys 0.016s
I20260619 06:35:52.660332 8244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:52.662438 8227 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.005s
I20260619 06:35:52.662721 8227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "4abed0a0edc345eb85aec2bbff01fe2e"
format_stamp: "Formatted at 2026-06-19 06:35:52 on dist-test-slave-nf4n"
server_key: "ca3111c302a8ccf60958bcfd002be684"
server_key_iv: "43cc156d9ca8fa2c714535df4bc573ee"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:52.663101 8227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:52.741140 8227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:52.746511 8227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:52.746910 8227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:52.754432 8227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:35:52.762760 8227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:35:52.762936 8227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 06:35:52.763130 8227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:35:52.763247 8227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:52.998104 8221 heartbeater.cc:503] Master 127.28.104.254:34397 was elected leader, sending a full tablet report...
I20260619 06:35:53.102716 8227 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:42291
I20260619 06:35:53.102720 8364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:42291 every 8 connection(s)
I20260619 06:35:53.104540 8227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:35:53.110267 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 8227
I20260619 06:35:53.110544 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:35:53.111243 29091 external_mini_cluster.cc:1428] Setting key e01b3be92882e6dc237296d72a01ccae
Jun 19 06:35:53 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850952, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
I20260619 06:35:53.147650 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.107780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:37633 (local address 127.28.104.254:34397)
0619 06:35:53.121435 (+ 13655us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:53.121445 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:53.122694 (+ 1249us) server_negotiation.cc:407] Connection header received
0619 06:35:53.127026 (+ 4332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:53.127036 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:53.127143 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:53.127231 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:53.129759 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.130970 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.133773 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.134551 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.134679 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.139685 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:53.139693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:53.139699 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:53.139752 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:53.143938 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:53.145014 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:53.145042 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:53.145048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:53.145208 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:53.145701 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:53.145706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:53.145709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:53.146109 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:53.146280 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:53.147035 (+ 755us) server_negotiation.cc:299] Negotiation successful
0619 06:35:53.147263 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13465,"thread_start_us":13665,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
I20260619 06:35:53.149631 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.108548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34397 (local address 127.28.104.194:37633)
0619 06:35:53.122185 (+ 13637us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:53.122259 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:53.126650 (+ 4391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:53.127860 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:53.127873 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:53.128638 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:35:53.129550 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.129579 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.131183 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.131191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.133603 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.133620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.135053 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.135063 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.135419 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.136252 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:35:53.136287 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:35:53.139387 (+ 3100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:35:53.144185 (+ 4798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:53.144197 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:53.144220 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:53.144860 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:53.145368 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:53.145371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:53.145373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:53.145565 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:53.146260 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:35:53.146268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:35:53.146641 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 06:35:53.146909 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 06:35:53.147985 (+ 1076us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13230,"thread_start_us":16343,"threads_started":1}
I20260619 06:35:53.156476 8365 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34397
I20260619 06:35:53.156836 8365 heartbeater.cc:464] Registering TS with master...
I20260619 06:35:53.157809 8365 heartbeater.cc:511] Master 127.28.104.254:34397 requested a full tablet report, sending...
I20260619 06:35:53.159543 8006 ts_manager.cc:194] Registered new tserver with Master: 4abed0a0edc345eb85aec2bbff01fe2e (127.28.104.194:42291)
I20260619 06:35:53.160802 8006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:37633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:35:53.178165 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:41177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34397
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:35:53.519466 8371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:35:53.519744 8371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:35:53.519917 8371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:35:53.527195 8371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:35:53.527302 8371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:35:53.527513 8371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:35:53.537827 8371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.104.212:41177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:35:53.539213 8371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:35:53.540540 8371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:35:53.554548 8380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:53.554601 8376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:35:53.554560 8377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:35:53.555631 8371 server_base.cc:1034] running on GCE node
I20260619 06:35:53.556334 8371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:35:53.557399 8371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:35:53.558657 8371 hybrid_clock.cc:648] HybridClock initialized: now 1781850953558630 us; error 76 us; skew 500 ppm
I20260619 06:35:53.559005 8371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:35:53 dist-test-slave-nf4n krb5kdc[7527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850953, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:35:53.562176 8371 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:35:53.563612 8371 webserver.cc:460] Webserver started at http://127.28.104.195:44345/ using document root <none> and password file <none>
I20260619 06:35:53.564244 8371 fs_manager.cc:359] Metadata directory not provided
I20260619 06:35:53.564339 8371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:35:53.564584 8371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:35:53 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850953, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:35:53.581751 8371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "cafd2f4af0294718b8659defb89251af"
format_stamp: "Formatted at 2026-06-19 06:35:53 on dist-test-slave-nf4n"
server_key: "2c5c3a8671c56e96428cc9d7b02143ce"
server_key_iv: "5fd09f477925fa4274160ac589b78f45"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:53.582623 8371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "cafd2f4af0294718b8659defb89251af"
format_stamp: "Formatted at 2026-06-19 06:35:53 on dist-test-slave-nf4n"
server_key: "2c5c3a8671c56e96428cc9d7b02143ce"
server_key_iv: "5fd09f477925fa4274160ac589b78f45"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:53.588357 8371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:35:53.605278 8387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:53.606915 8371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:35:53.607122 8371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "cafd2f4af0294718b8659defb89251af"
format_stamp: "Formatted at 2026-06-19 06:35:53 on dist-test-slave-nf4n"
server_key: "2c5c3a8671c56e96428cc9d7b02143ce"
server_key_iv: "5fd09f477925fa4274160ac589b78f45"
server_key_version: "kuduclusterkey@0"
I20260619 06:35:53.607427 8371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:35:53.629220 8371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:35:53.633901 8371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:35:53.634200 8371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:35:53.635226 8371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:35:53.636621 8371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:35:53.636722 8371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:53.636869 8371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:35:53.636940 8371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:35:53.673357 8371 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:35175
I20260619 06:35:53.673394 8499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:35175 every 8 connection(s)
I20260619 06:35:53.674626 8371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:35:53.681641 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 8371
I20260619 06:35:53.681809 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:35:53.682245 29091 external_mini_cluster.cc:1428] Setting key 067610ac5bef44bc68a6e3fd9a0b69e4
Jun 19 06:35:53 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850953, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:35:53.693372 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.677205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:54423 (local address 127.28.104.254:34397)
0619 06:35:53.677669 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:53.677673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:53.678539 (+ 866us) server_negotiation.cc:407] Connection header received
0619 06:35:53.680366 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:53.680372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:53.680457 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:53.680593 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:53.682123 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.682940 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.684489 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.684956 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.685111 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.688243 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:53.688248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:53.688252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:53.688286 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:53.690703 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:53.691337 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:53.691341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:53.691343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:53.691454 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:53.691821 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:53.691824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:53.691825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:53.692140 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:53.692244 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:53.692832 (+ 588us) server_negotiation.cc:299] Negotiation successful
0619 06:35:53.692985 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":209,"threads_started":1}
I20260619 06:35:53.694336 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.677704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34397 (local address 127.28.104.195:54423)
0619 06:35:53.678382 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:53.678424 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:53.680100 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:53.680826 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:53.680834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:53.681302 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:35:53.681964 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.681982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.683112 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.683117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.684346 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.684357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.685206 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.685214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.685445 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.686102 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:35:53.686123 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:35:53.688032 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:35:53.690832 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:53.690838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:53.690849 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:53.691245 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:53.691539 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:35:53.691542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:35:53.691544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:35:53.691736 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:35:53.692240 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:35:53.692245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:35:53.692541 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 06:35:53.692764 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 06:35:53.693290 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":271,"threads_started":1}
I20260619 06:35:53.695786 8500 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34397
I20260619 06:35:53.696121 8500 heartbeater.cc:464] Registering TS with master...
I20260619 06:35:53.696928 8500 heartbeater.cc:511] Master 127.28.104.254:34397 requested a full tablet report, sending...
I20260619 06:35:53.698406 8006 ts_manager.cc:194] Registered new tserver with Master: cafd2f4af0294718b8659defb89251af (127.28.104.195:35175)
I20260619 06:35:53.699227 8006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:54423
I20260619 06:35:53.709823 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:35:53.726649 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.714552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.28.104.254:34397)
0619 06:35:53.714716 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:53.714721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:53.715262 (+ 541us) server_negotiation.cc:407] Connection header received
0619 06:35:53.715420 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:53.715424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:53.715506 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:53.715589 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:53.716943 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.717729 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.719019 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.719434 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.719529 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.721885 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:53.721889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:53.721893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:53.721921 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:53.724338 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:53.724943 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:53.724946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:53.724949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:53.725088 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:53.725397 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:53.725401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:53.725403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:53.725669 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:53.725854 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:53.726188 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 06:35:53.726357 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:35:53.730731 8006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:35:53.738138 8006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:35:53.767980 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.750739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.28.104.194:42291)
0619 06:35:53.751261 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:53.751269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:53.751415 (+ 146us) server_negotiation.cc:407] Connection header received
0619 06:35:53.751772 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:53.751784 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:53.758122 (+ 6338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:53.758405 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:35:53.759722 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.761788 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.765345 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.766758 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.766928 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.767120 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:53.767511 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 06:35:53.767691 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":252,"threads_started":1}
I20260619 06:35:53.768110 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.750545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:42291 (local address 127.0.0.1:39826)
0619 06:35:53.751261 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:53.751306 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:53.751491 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:53.758426 (+ 6935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:53.758438 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:53.758482 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:35:53.759155 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.759172 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.762318 (+ 3146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.762323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.765187 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.765201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.766984 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.766990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.767208 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.767283 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 06:35:53.767432 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 06:35:53.767679 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":28,"thread_start_us":286,"threads_started":1}
I20260619 06:35:53.772222 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.751891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:35089 (local address 127.0.0.1:34464)
0619 06:35:53.763353 (+ 11462us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:53.763371 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:53.763484 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:53.764880 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:53.764885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:53.764907 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:35:53.765328 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.765338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.767189 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.767194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.769627 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.769641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.771478 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.771484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.771642 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.771675 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:35:53.771760 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:35:53.771875 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11335,"thread_start_us":10878,"threads_started":1}
I20260619 06:35:53.772326 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.752260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.28.104.193:35089)
0619 06:35:53.764006 (+ 11746us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:53.764016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:53.764045 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:35:53.764137 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:53.764145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:53.764400 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:53.764618 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:35:53.765486 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.767016 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.769781 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.771267 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.771446 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.771603 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:53.771847 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 06:35:53.771995 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11548,"thread_start_us":12000,"threads_started":1}
I20260619 06:35:53.772264 8292 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc1352f43d54e7085e211b0c35cf1e4 (DEFAULT_TABLE table=test-table [id=2670964b0594489b88df62ce681ff735]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:35:53.773211 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.751701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:35175 (local address 127.0.0.1:40086)
0619 06:35:53.762525 (+ 10824us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:53.762555 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:53.762682 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:53.765500 (+ 2818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:53.765503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:53.765527 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:35:53.765944 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.765956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.768665 (+ 2709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.768671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.770758 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.770770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.772530 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.772536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.772722 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.772744 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:35:53.772815 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 06:35:53.772892 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10660,"mutex_wait_us":1,"thread_start_us":11401,"threads_started":1}
I20260619 06:35:53.774117 8292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc1352f43d54e7085e211b0c35cf1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:35:53.775439 8154 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc1352f43d54e7085e211b0c35cf1e4 (DEFAULT_TABLE table=test-table [id=2670964b0594489b88df62ce681ff735]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:35:53.776727 8154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc1352f43d54e7085e211b0c35cf1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:35:53.777196 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.752109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.28.104.195:35175)
0619 06:35:53.764950 (+ 12841us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:53.764956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:53.764981 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:35:53.765101 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:53.765107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:53.765332 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:53.765512 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:35:53.766132 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.768528 (+ 2396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.770931 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.772194 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.773177 (+ 983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.776615 (+ 3438us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:53.776687 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 06:35:53.776849 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12676,"thread_start_us":12499,"threads_started":1}
I20260619 06:35:53.780586 8435 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc1352f43d54e7085e211b0c35cf1e4 (DEFAULT_TABLE table=test-table [id=2670964b0594489b88df62ce681ff735]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:35:53.782015 8435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc1352f43d54e7085e211b0c35cf1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:35:53.788898 8519 tablet_bootstrap.cc:492] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: Bootstrap starting.
I20260619 06:35:53.790913 8520 tablet_bootstrap.cc:492] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: Bootstrap starting.
I20260619 06:35:53.792809 8519 tablet_bootstrap.cc:654] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: Neither blocks nor log segments found. Creating new log.
I20260619 06:35:53.794103 8519 log.cc:824] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:35:53.794312 8520 tablet_bootstrap.cc:654] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: Neither blocks nor log segments found. Creating new log.
I20260619 06:35:53.795689 8520 log.cc:824] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:35:53.798803 8519 tablet_bootstrap.cc:492] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: No bootstrap required, opened a new log
I20260619 06:35:53.799137 8519 ts_tablet_manager.cc:1397] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 06:35:53.800904 8522 tablet_bootstrap.cc:492] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: Bootstrap starting.
I20260619 06:35:53.802783 8520 tablet_bootstrap.cc:492] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: No bootstrap required, opened a new log
I20260619 06:35:53.803048 8520 ts_tablet_manager.cc:1397] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260619 06:35:53.804425 8522 tablet_bootstrap.cc:654] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: Neither blocks nor log segments found. Creating new log.
I20260619 06:35:53.805647 8522 log.cc:824] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: Log is configured to *not* fsync() on all Append() calls
I20260619 06:35:53.808881 8522 tablet_bootstrap.cc:492] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: No bootstrap required, opened a new log
I20260619 06:35:53.809193 8522 ts_tablet_manager.cc:1397] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260619 06:35:53.809690 8519 raft_consensus.cc:348] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42291 } } peers { permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 } } peers { permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 } }
I20260619 06:35:53.810125 8519 raft_consensus.cc:374] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:35:53.810232 8519 raft_consensus.cc:729] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38922b7333743218b064758c736c6a8, State: Initialized, Role: FOLLOWER
I20260619 06:35:53.810865 8519 consensus_queue.cc:260] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42291 } } peers { permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 } } peers { permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 } }
I20260619 06:35:53.811600 8520 raft_consensus.cc:348] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42291 } } peers { permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 } } peers { permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 } }
I20260619 06:35:53.811988 8520 raft_consensus.cc:374] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:35:53.812098 8520 raft_consensus.cc:729] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4abed0a0edc345eb85aec2bbff01fe2e, State: Initialized, Role: FOLLOWER
I20260619 06:35:53.812825 8520 consensus_queue.cc:260] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42291 } } peers { permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 } } peers { permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 } }
I20260619 06:35:53.813894 8365 heartbeater.cc:503] Master 127.28.104.254:34397 was elected leader, sending a full tablet report...
I20260619 06:35:53.814601 8520 ts_tablet_manager.cc:1428] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 06:35:53.817632 8519 ts_tablet_manager.cc:1428] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260619 06:35:53.820215 8522 raft_consensus.cc:348] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42291 } } peers { permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 } } peers { permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 } }
I20260619 06:35:53.820541 8522 raft_consensus.cc:374] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:35:53.820611 8522 raft_consensus.cc:729] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cafd2f4af0294718b8659defb89251af, State: Initialized, Role: FOLLOWER
I20260619 06:35:53.821117 8522 consensus_queue.cc:260] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42291 } } peers { permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 } } peers { permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 } }
I20260619 06:35:53.822037 8500 heartbeater.cc:503] Master 127.28.104.254:34397 was elected leader, sending a full tablet report...
I20260619 06:35:53.822530 8522 ts_tablet_manager.cc:1428] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260619 06:35:53.876744 8366 tablet.cc:2369] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:35:53.926216 8501 tablet.cc:2369] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:35:53.960403 8222 tablet.cc:2369] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:35:53.989454 8526 raft_consensus.cc:482] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:35:53.989900 8526 raft_consensus.cc:504] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42291 } } peers { permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 } } peers { permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 } }
I20260619 06:35:53.991641 8526 leader_election.cc:288] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cafd2f4af0294718b8659defb89251af (127.28.104.195:35175), f38922b7333743218b064758c736c6a8 (127.28.104.193:35089)
I20260619 06:35:53.998135 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.991917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:57969 (local address 127.28.104.195:35175)
0619 06:35:53.992124 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:53.992133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:53.992823 (+ 690us) server_negotiation.cc:407] Connection header received
0619 06:35:53.993071 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:53.993078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:53.993173 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:53.993284 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:35:53.993839 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.994536 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.996377 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.997215 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.997390 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.997468 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:53.997755 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 06:35:53.997872 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:35:53.998240 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.991860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:35175 (local address 127.28.104.194:57969)
0619 06:35:53.992700 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:53.992720 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:53.992898 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:53.993299 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:53.993303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:53.993334 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:35:53.993705 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.993713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.994664 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.994671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.996231 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.996242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.997415 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.997422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.997595 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.997621 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:35:53.997730 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 06:35:53.997852 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":527,"threads_started":1}
I20260619 06:35:53.998245 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.992202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:34109 (local address 127.28.104.193:35089)
0619 06:35:53.992411 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:53.992420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:53.993038 (+ 618us) server_negotiation.cc:407] Connection header received
0619 06:35:53.993321 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:53.993324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:53.993381 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:53.993475 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:35:53.994051 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.994755 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.996530 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.997332 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.997506 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.997579 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:53.997849 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:35:53.997971 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:35:53.998288 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:53.992109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:35089 (local address 127.28.104.194:34109)
0619 06:35:53.992898 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0619 06:35:53.992912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:35:53.993150 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:35:53.993506 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:35:53.993510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:35:53.993533 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:35:53.993890 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.993898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.994884 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.994894 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.996379 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:35:53.996390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:35:53.997547 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:35:53.997553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:35:53.997704 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:53.997725 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:35:53.997824 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:35:53.997940 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":530,"threads_started":1}
I20260619 06:35:53.999298 8174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc1352f43d54e7085e211b0c35cf1e4" candidate_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f38922b7333743218b064758c736c6a8" is_pre_election: true
I20260619 06:35:53.999302 8455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc1352f43d54e7085e211b0c35cf1e4" candidate_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cafd2f4af0294718b8659defb89251af" is_pre_election: true
I20260619 06:35:53.999922 8174 raft_consensus.cc:2454] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4abed0a0edc345eb85aec2bbff01fe2e in term 0.
I20260619 06:35:53.999994 8455 raft_consensus.cc:2454] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4abed0a0edc345eb85aec2bbff01fe2e in term 0.
I20260619 06:35:54.000638 8248 leader_election.cc:302] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4abed0a0edc345eb85aec2bbff01fe2e, f38922b7333743218b064758c736c6a8; no voters:
I20260619 06:35:54.001125 8526 raft_consensus.cc:2781] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:35:54.001288 8526 raft_consensus.cc:482] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:35:54.001387 8526 raft_consensus.cc:3037] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:35:54.003535 8526 raft_consensus.cc:504] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42291 } } peers { permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 } } peers { permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 } }
I20260619 06:35:54.004109 8526 leader_election.cc:288] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [CANDIDATE]: Term 1 election: Requested vote from peers cafd2f4af0294718b8659defb89251af (127.28.104.195:35175), f38922b7333743218b064758c736c6a8 (127.28.104.193:35089)
I20260619 06:35:54.004515 8174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc1352f43d54e7085e211b0c35cf1e4" candidate_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f38922b7333743218b064758c736c6a8"
I20260619 06:35:54.004519 8455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc1352f43d54e7085e211b0c35cf1e4" candidate_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cafd2f4af0294718b8659defb89251af"
I20260619 06:35:54.004698 8174 raft_consensus.cc:3037] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:35:54.004714 8455 raft_consensus.cc:3037] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:35:54.006865 8174 raft_consensus.cc:2454] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4abed0a0edc345eb85aec2bbff01fe2e in term 1.
I20260619 06:35:54.007105 8455 raft_consensus.cc:2454] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4abed0a0edc345eb85aec2bbff01fe2e in term 1.
I20260619 06:35:54.007267 8248 leader_election.cc:302] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4abed0a0edc345eb85aec2bbff01fe2e, f38922b7333743218b064758c736c6a8; no voters:
I20260619 06:35:54.007557 8526 raft_consensus.cc:2781] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:35:54.008136 8526 raft_consensus.cc:686] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 1 LEADER]: Becoming Leader. State: Replica: 4abed0a0edc345eb85aec2bbff01fe2e, State: Running, Role: LEADER
I20260619 06:35:54.008680 8526 consensus_queue.cc:237] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42291 } } peers { permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 } } peers { permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 } }
I20260619 06:35:54.013602 8005 catalog_manager.cc:5495] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4abed0a0edc345eb85aec2bbff01fe2e (127.28.104.194). New cstate: current_term: 1 leader_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abed0a0edc345eb85aec2bbff01fe2e" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 42291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 } health_report { overall_health: UNKNOWN } } }
I20260619 06:35:54.070130 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:54.064912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.28.104.194:42291)
0619 06:35:54.065165 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:54.065169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:54.065187 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:35:54.065271 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:54.065276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:54.065357 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:54.065483 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:35:54.065797 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:54.066545 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:54.067839 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:54.068269 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:54.068403 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:54.068927 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:35:54.069566 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:35:54.069722 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:54.069776 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 06:35:54.069848 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:35:54.078007 8174 raft_consensus.cc:1261] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8 [term 1 FOLLOWER]: Refusing update from remote peer 4abed0a0edc345eb85aec2bbff01fe2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:35:54.078014 8455 raft_consensus.cc:1261] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af [term 1 FOLLOWER]: Refusing update from remote peer 4abed0a0edc345eb85aec2bbff01fe2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:35:54.078958 8531 consensus_queue.cc:1035] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cafd2f4af0294718b8659defb89251af" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:35:54.079277 8526 consensus_queue.cc:1035] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f38922b7333743218b064758c736c6a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 35089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:35:54.089854 8534 mvcc.cc:205] Tried to move back new op lower bound from 7298461507894276096 to 7298461507623755776. Current Snapshot: MvccSnapshot[applied={T|T < 7298461507894276096}]
I20260619 06:35:54.092144 8535 mvcc.cc:205] Tried to move back new op lower bound from 7298461507894276096 to 7298461507623755776. Current Snapshot: MvccSnapshot[applied={T|T < 7298461507894276096}]
I20260619 06:35:54.096112 8536 mvcc.cc:205] Tried to move back new op lower bound from 7298461507894276096 to 7298461507623755776. Current Snapshot: MvccSnapshot[applied={T|T < 7298461507894276096}]
I20260619 06:35:54.104836 8005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:35:54.105134 8005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:35:54.108485 8005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ce1e55ef3d34a4cbe3654229e2cc729: Sending DeleteTablet for 3 replicas of tablet 3fc1352f43d54e7085e211b0c35cf1e4
I20260619 06:35:54.109529 8292 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc1352f43d54e7085e211b0c35cf1e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:35:54 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:39826
I20260619 06:35:54.109716 8154 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc1352f43d54e7085e211b0c35cf1e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:35:54 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:34464
I20260619 06:35:54.110742 8435 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc1352f43d54e7085e211b0c35cf1e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:35:54 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:40086
I20260619 06:35:54.114351 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 8086
I20260619 06:35:54.122521 8545 tablet_replica.cc:330] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: stopping tablet replica
I20260619 06:35:54.123049 8545 raft_consensus.cc:2229] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:35:54.123605 8545 raft_consensus.cc:2258] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:35:54.126351 8544 tablet_replica.cc:330] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: stopping tablet replica
I20260619 06:35:54.126828 8544 raft_consensus.cc:2229] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:35:54.127225 8545 ts_tablet_manager.cc:1905] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:35:54.127362 8544 raft_consensus.cc:2258] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:35:54.134392 8545 ts_tablet_manager.cc:1918] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:35:54.134552 8545 log.cc:1196] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/3fc1352f43d54e7085e211b0c35cf1e4
I20260619 06:35:54.134959 8545 ts_tablet_manager.cc:1939] T 3fc1352f43d54e7085e211b0c35cf1e4 P cafd2f4af0294718b8659defb89251af: Deleting consensus metadata
I20260619 06:35:54.136489 7990 catalog_manager.cc:4841] TS cafd2f4af0294718b8659defb89251af (127.28.104.195:35175): tablet 3fc1352f43d54e7085e211b0c35cf1e4 (table test-table [id=2670964b0594489b88df62ce681ff735]) successfully deleted
I20260619 06:35:54.137382 8543 tablet_replica.cc:330] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: stopping tablet replica
I20260619 06:35:54.137804 8543 raft_consensus.cc:2229] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:35:54.138418 8543 raft_consensus.cc:2258] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:35:54.139206 8544 ts_tablet_manager.cc:1905] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:35:54.140504 8543 ts_tablet_manager.cc:1905] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:35:54.146438 8544 ts_tablet_manager.cc:1918] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:35:54.146624 8544 log.cc:1196] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/3fc1352f43d54e7085e211b0c35cf1e4
I20260619 06:35:54.147145 8544 ts_tablet_manager.cc:1939] T 3fc1352f43d54e7085e211b0c35cf1e4 P f38922b7333743218b064758c736c6a8: Deleting consensus metadata
I20260619 06:35:54.148095 8543 ts_tablet_manager.cc:1918] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 19 06:35:54 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:35:54.148319 8543 log.cc:1196] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/3fc1352f43d54e7085e211b0c35cf1e4
I20260619 06:35:54.148847 8543 ts_tablet_manager.cc:1939] T 3fc1352f43d54e7085e211b0c35cf1e4 P 4abed0a0edc345eb85aec2bbff01fe2e: Deleting consensus metadata
I20260619 06:35:54.149017 7993 catalog_manager.cc:4841] TS f38922b7333743218b064758c736c6a8 (127.28.104.193:35089): tablet 3fc1352f43d54e7085e211b0c35cf1e4 (table test-table [id=2670964b0594489b88df62ce681ff735]) successfully deleted
I20260619 06:35:54.150440 7990 catalog_manager.cc:4841] TS 4abed0a0edc345eb85aec2bbff01fe2e (127.28.104.194:42291): tablet 3fc1352f43d54e7085e211b0c35cf1e4 (table test-table [id=2670964b0594489b88df62ce681ff735]) successfully deleted
I20260619 06:35:54.155936 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:54.115547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.28.104.193:35089)
0619 06:35:54.115715 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:54.115721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:54.137663 (+ 21942us) server_negotiation.cc:407] Connection header received
0619 06:35:54.137896 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:54.137901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:54.137983 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:54.138101 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:54.139953 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:54.141067 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:54.143151 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:54.143859 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:54.143968 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:54.149961 (+ 5993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:54.149974 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:54.149998 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:54.150029 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:54.153240 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:54.153917 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:54.153926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:54.153930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:54.154060 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:54.154425 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:54.154431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:54.154434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:54.154784 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:54.155083 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:54.155381 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 06:35:54.155578 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:35:54.156559 8216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:54.246873 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 8086
I20260619 06:35:54.264945 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 8227
Jun 19 06:35:54 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:35:54.281396 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:54.265875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.28.104.194:42291)
0619 06:35:54.266052 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:54.266060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:54.266083 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:35:54.266311 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:54.266317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:54.266399 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:54.266517 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:54.267806 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:54.268914 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:54.270355 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:54.270807 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:54.270962 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:54.275546 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:54.275559 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:54.275575 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:54.275612 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:54.278666 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:54.279318 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:54.279325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:54.279327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:54.279469 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:54.279831 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:54.279836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:54.279847 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:54.280181 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:54.280516 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:54.280790 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 06:35:54.281009 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:35:54.282553 8360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:54.395131 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 8227
I20260619 06:35:54.413813 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 8371
Jun 19 06:35:54 dist-test-slave-nf4n krb5kdc[7527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:35:54.429163 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:35:54.414664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.28.104.195:35175)
0619 06:35:54.414875 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:35:54.414881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:35:54.414902 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:35:54.415103 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:35:54.415116 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:35:54.415178 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:35:54.415285 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:35:54.416673 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:54.417482 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:54.418850 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:35:54.419257 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:35:54.419394 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:35:54.423691 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:35:54.423707 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:35:54.423724 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:35:54.423760 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:35:54.426432 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:54.427141 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:54.427148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:54.427150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:54.427285 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:35:54.427763 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:35:54.427766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:35:54.427768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:35:54.428147 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:35:54.428344 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 06:35:54.428605 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 06:35:54.428755 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:35:54.429788 8495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:54.517668 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 8371
I20260619 06:35:54.534384 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 7974
I20260619 06:35:54.535604 8036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:35:54.648663 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 7974
I20260619 06:35:54.674139 29091 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:35:55.019798 29091 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:35:55.020093 29091 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:35:55.365880 29091 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:35:55.366 UTC [7557] LOG: received smart shutdown request
2026-06-19 06:35:55.369 UTC [7557] LOG: background worker "logical replication launcher" (PID 7566) exited with exit code 1
2026-06-19 06:35:55.370 UTC [7561] LOG: shutting down
2026-06-19 06:35:55.537 UTC [7557] LOG: database system is shut down
2026-06-19T06:35:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48370 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:35:55 dist-test-slave-nf4n krb5kdc[8559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:35:55 dist-test-slave-nf4n krb5kdc[8559](info): set up 2 sockets
Jun 19 06:35:55 dist-test-slave-nf4n krb5kdc[8559](info): commencing operation
krb5kdc: starting...
W20260619 06:35:56.801936 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:35:56 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:35:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:35:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.104.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.104.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/rangeradmin_127.28.104.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/rangeradmin_127.28.104.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.104.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.104.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/rangerlookup_127.28.104.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/rangerlookup_127.28.104.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.104.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.104.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/HTTP_127.28.104.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/HTTP_127.28.104.212@KRBTEST.COM.keytab.
I20260619 06:35:57.120827 29091 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskS2JAv5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/postgres -l logfile start
2026-06-19 06:35:59.624 UTC [8589] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:35:59.624 UTC [8589] LOG: listening on IPv4 address "127.28.104.212", port 32907
2026-06-19 06:35:59.628 UTC [8589] LOG: listening on Unix socket "/tmp/.s.PGSQL.32907"
2026-06-19 06:35:59.651 UTC [8592] LOG: database system was shut down at 2026-06-19 06:35:58 UTC
2026-06-19 06:35:59.655 UTC [8589] LOG: database system is ready to accept connections
I20260619 06:36:00.628785 29091 mini_postgres.cc:96] Postgres bound to 32907
2026-06-19 06:36:00.645 UTC [8600] FATAL: database "slave" does not exist
127.28.104.212:32907 - accepting connections
I20260619 06:36:00.646540 29091 mini_ranger.cc:161] Starting Ranger...
I20260619 06:36:00.672267 29091 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:36:01.121196 29091 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:36:01.121315 29091 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-admin
2026-06-19 06:36:01,404 [I] DB FLAVOR :POSTGRES
2026-06-19 06:36:01,404 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:36:01,404 [I] Checking connection..
2026-06-19 06:36:01,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:36:01,662 [I] Checking connection passed.
2026-06-19 06:36:01,663 [I] --------- Verifying version history table ---------
2026-06-19 06:36:01,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:36:01,929 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:36:01,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:36:02,170 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:36:02,170 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:36:02,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:36:02.395 UTC [8664] WARNING: there is no transaction in progress
2026-06-19 06:36:02,401 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:36:02,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:36:02,655 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:36:02,655 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:36:02,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:36:02,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:36:03,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nf4n.c.gcp-upstream.internal','N') ;"
2026-06-19 06:36:03,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:36:03,633 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:36:03,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:36:03,881 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:36:03,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:36:04,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:36:04,369 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:36:04,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:36:05.325 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.341 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.357 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.372 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.387 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.404 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.465 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.472 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.486 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.497 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.508 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.521 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.531 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.537 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.544 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:05.549 UTC [8773] WARNING: there is no transaction in progress
2026-06-19 06:36:06,151 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:36:06,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:36:06,401 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:36:06,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:36:06,668 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:36:06,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:36:06,901 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:36:06,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:36:07,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:36:07,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nf4n.c.gcp-upstream.internal';"
2026-06-19 06:36:07,623 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:36:07,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:36:07,871 [I] DB_PATCHES have already been applied
I20260619 06:36:07.877111 29091 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-admin:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskS2JAv5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:36:07.877194 29091 mini_ranger.cc:193] Using host: 127.28.104.212
I20260619 06:36:07.891891 29091 mini_ranger.cc:227] Ranger admin URL: http://127.28.104.212:47573
Jun 19, 2026 6:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:36:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:36:09.030452 29091 mini_ranger.cc:160] Time spent starting Ranger: real 8.384s user 0.006s sys 0.031s
Jun 19, 2026 6:36:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:36:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 6:36:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:36:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:36:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:36:31.397327 29091 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.104.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.104.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/rangerkms_127.28.104.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.104.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/rangerkms_127.28.104.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.104.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.104.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.104.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/HTTP_127.28.104.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.104.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/HTTP_127.28.104.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:36:31 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850991, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:36:31.563484 29091 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:36:31.584050 29091 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:36:32.379920 29091 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:36:32.380062 29091 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-kms
I20260619 06:36:32.380124 29091 mini_ranger_kms.cc:227] Using postgres at 127.28.104.212:32907
2026-06-19 06:36:32,733 [I] DB FLAVOR :POSTGRES
2026-06-19 06:36:32,733 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:36:32,733 [I] Checking connection
2026-06-19 06:36:32,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:36:32,975 [I] connection success
2026-06-19 06:36:32,975 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:36:32,975 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:36:32,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:36:33,219 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:36:33,219 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:36:33,219 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:36:33,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.104.212:32907/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:36:33,471 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:36:33.967109 29091 mini_ranger_kms.cc:310] Created kms service
I20260619 06:36:34.151897 29091 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:36:34.199930 29091 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:36:34 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850991, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:36:34.629106 29091 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:36:34.629462 29091 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-kms:/tmp/dist-test-taskS2JAv5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskS2JAv5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:36:34.629519 29091 mini_ranger_kms.cc:240] Using host: 127.28.104.212
I20260619 06:36:34.647660 29091 mini_ranger_kms.cc:276] Ranger KMS PID: 8954
I20260619 06:36:34.647818 29091 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.104.212:43429
Jun 19, 2026 6:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:36:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:36:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:36:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 06:36:36.972054 29091 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.409s user 0.068s sys 0.298s
I20260619 06:36:36.972316 29091 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:36:36.972483 29091 mini_ranger_kms.cc:401] 127.28.104.212:43429/kms/v1/keys
Jun 19 06:36:37 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781850997, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.104.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 06:36:37 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850997, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.104.212@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
Jun 19 06:36:37 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850997, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.104.212@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
Jun 19 06:36:40 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:36:40.272939 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:44423
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:44423
--ranger_config_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:43429/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:36:40.614585 9005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:40.614965 9005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:40.615149 9005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:40.622601 9005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:36:40.622774 9005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:40.622867 9005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:40.623005 9005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:36:40.623099 9005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:36:40.634435 9005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.104.212:43429/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:44423
--ranger_config_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:44423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:40.635845 9005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:40.637362 9005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:40.649786 9013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:40.649868 9010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:40.649868 9011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:40.650993 9005 server_base.cc:1034] running on GCE node
I20260619 06:36:40.651818 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:40.653473 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:40.654736 9005 hybrid_clock.cc:648] HybridClock initialized: now 1781851000654686 us; error 32 us; skew 500 ppm
I20260619 06:36:40.655121 9005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:40 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:40.658449 9005 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:36:40.660048 9005 webserver.cc:460] Webserver started at http://127.28.104.254:44221/ using document root <none> and password file <none>
I20260619 06:36:40.660750 9005 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:40.660902 9005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:40.661288 9005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:36:40 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:36:40.718787 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "d6506013e29d48a4be04f75370fb7646"
format_stamp: "Formatted at 2026-06-19 06:36:40 on dist-test-slave-nf4n"
server_key: "a422f20397fa6e1f23641bded6c16ce2"
server_key_iv: "a6898035a418bcb521347f7dd3142499"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:40.719947 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "d6506013e29d48a4be04f75370fb7646"
format_stamp: "Formatted at 2026-06-19 06:36:40 on dist-test-slave-nf4n"
server_key: "a422f20397fa6e1f23641bded6c16ce2"
server_key_iv: "a6898035a418bcb521347f7dd3142499"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:40.727144 9005 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260619 06:36:40.747557 9020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:40.749342 9005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 06:36:40.749513 9005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "d6506013e29d48a4be04f75370fb7646"
format_stamp: "Formatted at 2026-06-19 06:36:40 on dist-test-slave-nf4n"
server_key: "a422f20397fa6e1f23641bded6c16ce2"
server_key_iv: "a6898035a418bcb521347f7dd3142499"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:40.749732 9005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:40.787985 9005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:40.791929 9005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:40.792191 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:40.810534 9005 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:44423
I20260619 06:36:40.810530 9073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:44423 every 8 connection(s)
I20260619 06:36:40.811913 9005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:36:40.815636 9074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:40.817281 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 9005
I20260619 06:36:40.817524 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:36:40.818315 29091 external_mini_cluster.cc:1428] Setting key 8e08d829bdd04435094e31f4fceb46c8
I20260619 06:36:40.827899 9074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: Bootstrap starting.
I20260619 06:36:40.831769 9074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:40.833096 9074 log.cc:824] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:36:40 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:40.836838 9074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: No bootstrap required, opened a new log
I20260619 06:36:40.843034 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:40.820438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.28.104.254:44423)
0619 06:36:40.821400 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:40.821415 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:40.821456 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:36:40.823080 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:40.823107 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:40.823486 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:40.823818 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:40.825470 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:40.826779 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:40.828630 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:40.829433 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:40.829574 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:40.834726 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:40.834735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:40.834752 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:40.834789 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:40.838650 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:40.839407 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:40.839417 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:40.839436 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:40.839604 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:40.840044 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:40.840050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:40.840054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:40.840627 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:40.840936 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:40.841536 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 06:36:40.841933 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":431,"threads_started":1}
I20260619 06:36:40.845400 9074 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } }
I20260619 06:36:40.845769 9074 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:40.845886 9074 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6506013e29d48a4be04f75370fb7646, State: Initialized, Role: FOLLOWER
I20260619 06:36:40.846521 9074 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } }
I20260619 06:36:40.846761 9074 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:36:40.846906 9074 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:36:40.847083 9074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:40.849334 9074 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } }
I20260619 06:36:40.849803 9074 leader_election.cc:302] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6506013e29d48a4be04f75370fb7646; no voters:
I20260619 06:36:40.850355 9074 leader_election.cc:288] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:36:40.850482 9079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:36:40.851188 9079 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 1 LEADER]: Becoming Leader. State: Replica: d6506013e29d48a4be04f75370fb7646, State: Running, Role: LEADER
I20260619 06:36:40.851701 9079 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } }
I20260619 06:36:40.852535 9074 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:36:40.855098 9080 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6506013e29d48a4be04f75370fb7646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } } }
I20260619 06:36:40.855047 9081 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6506013e29d48a4be04f75370fb7646. Latest consensus state: current_term: 1 leader_uuid: "d6506013e29d48a4be04f75370fb7646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } } }
I20260619 06:36:40.855515 9080 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [sys.catalog]: This master's current role is: LEADER
I20260619 06:36:40.855515 9081 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [sys.catalog]: This master's current role is: LEADER
I20260619 06:36:40.857373 9086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:36:40.862033 9086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:36:40.866511 9074 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:36:40.877111 9086 catalog_manager.cc:1347] Generated new cluster ID: c07d5ba05e5647e399c966cab22bce1f
I20260619 06:36:40.877332 9086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:36:40.907167 9086 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:36:40.909343 9086 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:36:40.920405 9086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: Generated new TSK 0
I20260619 06:36:40.921535 9086 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:36:41.656490 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:43429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44423
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:36:41 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851001, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:36:42.060073 9119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:42.060454 9119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:42.060657 9119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:42.072211 9119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:42.072423 9119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:42.072631 9119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:36:42.087340 9119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.104.212:43429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:42.089159 9119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:42.090830 9119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:42.113308 9129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:42.114398 9130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:42.123152 9119 server_base.cc:1034] running on GCE node
W20260619 06:36:42.123428 9133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:42.124442 9119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:42.125856 9119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:42.127199 9119 hybrid_clock.cc:648] HybridClock initialized: now 1781851002127121 us; error 48 us; skew 500 ppm
I20260619 06:36:42.127650 9119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:42 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851002, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:42.135857 9119 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:36:42.138026 9119 webserver.cc:460] Webserver started at http://127.28.104.193:35101/ using document root <none> and password file <none>
I20260619 06:36:42.139009 9119 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:42.139153 9119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:42.139547 9119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:36:42 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851002, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:36:42.160730 9119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "84d26332369b429e967da7b700112790"
format_stamp: "Formatted at 2026-06-19 06:36:42 on dist-test-slave-nf4n"
server_key: "495c771e2e1a980d0756ffc2d601c794"
server_key_iv: "1aa216e460ffa3b65e507247d698926a"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:42.161868 9119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "84d26332369b429e967da7b700112790"
format_stamp: "Formatted at 2026-06-19 06:36:42 on dist-test-slave-nf4n"
server_key: "495c771e2e1a980d0756ffc2d601c794"
server_key_iv: "1aa216e460ffa3b65e507247d698926a"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:42.195708 9119 fs_manager.cc:691] Time spent creating directory manager: real 0.033s user 0.021s sys 0.011s
I20260619 06:36:42.215981 9140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:42.218060 9119 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 06:36:42.218386 9119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "84d26332369b429e967da7b700112790"
format_stamp: "Formatted at 2026-06-19 06:36:42 on dist-test-slave-nf4n"
server_key: "495c771e2e1a980d0756ffc2d601c794"
server_key_iv: "1aa216e460ffa3b65e507247d698926a"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:42.218710 9119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:42.250645 9119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:42.255790 9119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:42.256196 9119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:42.262352 9119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:36:42.266942 9119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:36:42.267052 9119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:42.267341 9119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:36:42.267498 9119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:42.339408 9119 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:45191
I20260619 06:36:42.341468 9119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:36:42.339419 9253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:45191 every 8 connection(s)
I20260619 06:36:42.347322 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 9119
I20260619 06:36:42.347572 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:36:42.348577 29091 external_mini_cluster.cc:1428] Setting key 63765d340430b2272d7cd5e8fc2bedbe
Jun 19 06:36:42 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851002, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
I20260619 06:36:42.381743 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:42.345073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:59179 (local address 127.28.104.254:44423)
0619 06:36:42.357478 (+ 12405us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:42.357487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:42.358749 (+ 1262us) server_negotiation.cc:407] Connection header received
0619 06:36:42.362757 (+ 4008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:42.362764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:42.362869 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:42.362991 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:42.365282 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:42.366459 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:42.368880 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:42.369561 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:42.369670 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:42.374318 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:42.374325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:42.374330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:42.374371 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:42.377996 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:42.378990 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:42.378996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:42.379000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:42.379151 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:42.379695 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:42.379700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:42.379703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:42.380029 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:42.380152 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:42.381124 (+ 972us) server_negotiation.cc:299] Negotiation successful
0619 06:36:42.381354 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12226,"thread_start_us":12068,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
I20260619 06:36:42.384232 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:42.345970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:44423 (local address 127.28.104.193:59179)
0619 06:36:42.358119 (+ 12149us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:42.358182 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:42.362370 (+ 4188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:42.363487 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:42.363500 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:42.364161 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:42.364987 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:42.365025 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:42.366663 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:42.366668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:42.368711 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:42.368732 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:42.370019 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:42.370026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:42.370318 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:42.371136 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:42.371167 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:42.374034 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:42.378189 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:42.378212 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:42.378231 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:42.378851 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:42.379301 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:42.379306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:42.379309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:42.379574 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:42.380258 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:42.380267 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:42.380687 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 06:36:42.382275 (+ 1588us) client_negotiation.cc:240] Negotiation successful
0619 06:36:42.382849 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11819,"thread_start_us":11559,"threads_started":1}
I20260619 06:36:42.386399 9254 heartbeater.cc:347] Connected to a master server at 127.28.104.254:44423
I20260619 06:36:42.386708 9254 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:42.387569 9254 heartbeater.cc:511] Master 127.28.104.254:44423 requested a full tablet report, sending...
I20260619 06:36:42.390133 9038 ts_manager.cc:194] Registered new tserver with Master: 84d26332369b429e967da7b700112790 (127.28.104.193:45191)
I20260619 06:36:42.392182 9038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:59179
Jun 19 06:36:42 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851001, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:36:42.430896 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:43429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44423
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:36:42 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851001, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
W20260619 06:36:42.875162 9260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:42.875739 9260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:42.875957 9260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:42.887854 9260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:42.888088 9260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:42.888327 9260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:36:42.904000 9260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.104.212:43429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:42.906378 9260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:42.908253 9260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:42.942270 9268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:42.946209 9267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:42.948915 9271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:42.949457 9260 server_base.cc:1034] running on GCE node
I20260619 06:36:42.950423 9260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:42.955181 9260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:42.956638 9260 hybrid_clock.cc:648] HybridClock initialized: now 1781851002956525 us; error 115 us; skew 500 ppm
I20260619 06:36:42.957140 9260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:42 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851002, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:42.961714 9260 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:36:42.976485 9260 webserver.cc:460] Webserver started at http://127.28.104.194:42561/ using document root <none> and password file <none>
I20260619 06:36:42.977605 9260 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:42.977852 9260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:42.978319 9260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:36:42 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851002, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:36:43.002416 9260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "2c78d0d9d6194a9ab43487e54d1dbb19"
format_stamp: "Formatted at 2026-06-19 06:36:42 on dist-test-slave-nf4n"
server_key: "ef23f91b481473e7492c0c15435b9362"
server_key_iv: "3fec6e86c7b66cdc5c1699cd19c9e3b6"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:43.003623 9260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "2c78d0d9d6194a9ab43487e54d1dbb19"
format_stamp: "Formatted at 2026-06-19 06:36:42 on dist-test-slave-nf4n"
server_key: "ef23f91b481473e7492c0c15435b9362"
server_key_iv: "3fec6e86c7b66cdc5c1699cd19c9e3b6"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:43.017360 9260 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.005s
I20260619 06:36:43.053939 9277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:43.059250 9260 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.000s sys 0.006s
I20260619 06:36:43.059469 9260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "2c78d0d9d6194a9ab43487e54d1dbb19"
format_stamp: "Formatted at 2026-06-19 06:36:42 on dist-test-slave-nf4n"
server_key: "ef23f91b481473e7492c0c15435b9362"
server_key_iv: "3fec6e86c7b66cdc5c1699cd19c9e3b6"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:43.059743 9260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:43.126341 9260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:43.130501 9260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:43.130812 9260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:43.131814 9260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:36:43.133323 9260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:36:43.133395 9260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:43.133541 9260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:36:43.133598 9260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:43.179517 9260 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:40693
I20260619 06:36:43.179636 9397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:40693 every 8 connection(s)
I20260619 06:36:43.180974 9260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:36:43.191397 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 9260
I20260619 06:36:43.191607 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:36:43.192183 29091 external_mini_cluster.cc:1428] Setting key c509d331623e59cd6306263f6971b948
Jun 19 06:36:43 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851002, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:43.207165 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:43.184087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:54685 (local address 127.28.104.254:44423)
0619 06:36:43.184578 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:43.184583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:43.185745 (+ 1162us) server_negotiation.cc:407] Connection header received
0619 06:36:43.187574 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:43.187580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:43.187695 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:43.187829 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:43.189350 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.190079 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.192013 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.192671 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.192795 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:43.197180 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:43.197189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:43.197197 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:43.197245 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:43.203011 (+ 5766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:43.204028 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:43.204038 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:43.204044 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:43.204207 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:43.204775 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:43.204783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:43.204788 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:43.205250 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:43.205409 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:43.206366 (+ 957us) server_negotiation.cc:299] Negotiation successful
0619 06:36:43.206635 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":279,"threads_started":1}
I20260619 06:36:43.208184 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:43.184757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:44423 (local address 127.28.104.194:54685)
0619 06:36:43.185543 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:43.185592 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:43.187298 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:43.188073 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:43.188082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:43.188526 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:43.189184 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:43.189203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.190243 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.190248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:43.191861 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:43.191876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.192905 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.192932 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:43.193274 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:43.194077 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:43.194120 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:43.196900 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:43.203209 (+ 6309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:43.203221 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:43.203241 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:43.203865 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:43.204336 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:43.204341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:43.204345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:43.204637 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:43.205405 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:43.205414 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:43.205903 (+ 489us) client_negotiation.cc:769] Sending connection context
0619 06:36:43.206234 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 06:36:43.206786 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":330,"threads_started":1}
I20260619 06:36:43.209759 9398 heartbeater.cc:347] Connected to a master server at 127.28.104.254:44423
I20260619 06:36:43.210088 9398 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:43.210953 9398 heartbeater.cc:511] Master 127.28.104.254:44423 requested a full tablet report, sending...
I20260619 06:36:43.212723 9038 ts_manager.cc:194] Registered new tserver with Master: 2c78d0d9d6194a9ab43487e54d1dbb19 (127.28.104.194:40693)
I20260619 06:36:43.213827 9038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:54685
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:36:43.248497 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:43429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44423
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:36:43.396031 9254 heartbeater.cc:503] Master 127.28.104.254:44423 was elected leader, sending a full tablet report...
W20260619 06:36:43.585493 9404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:43.585786 9404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:43.585947 9404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:43.593351 9404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:43.593464 9404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:43.593665 9404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:36:43.603966 9404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.104.212:43429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:44423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:43.605474 9404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:43.606813 9404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:43.620710 9412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:43.620606 9409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:43.620635 9410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:43.621559 9404 server_base.cc:1034] running on GCE node
I20260619 06:36:43.622215 9404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:43.623272 9404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:43.624496 9404 hybrid_clock.cc:648] HybridClock initialized: now 1781851003624412 us; error 71 us; skew 500 ppm
I20260619 06:36:43.624955 9404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:43 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:43.628408 9404 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:36:43.630108 9404 webserver.cc:460] Webserver started at http://127.28.104.195:41145/ using document root <none> and password file <none>
I20260619 06:36:43.630861 9404 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:43.631011 9404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:43.631328 9404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:36:43 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:36:43.650378 9404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4"
format_stamp: "Formatted at 2026-06-19 06:36:43 on dist-test-slave-nf4n"
server_key: "d569b3ff6f96040cbe3372478c4b493e"
server_key_iv: "6d996975d21bc576ee55991e108091dc"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:43.651816 9404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4"
format_stamp: "Formatted at 2026-06-19 06:36:43 on dist-test-slave-nf4n"
server_key: "d569b3ff6f96040cbe3372478c4b493e"
server_key_iv: "6d996975d21bc576ee55991e108091dc"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:43.663018 9404 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.010s sys 0.000s
I20260619 06:36:43.685671 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:43.687381 9404 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.007s sys 0.000s
I20260619 06:36:43.687589 9404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4"
format_stamp: "Formatted at 2026-06-19 06:36:43 on dist-test-slave-nf4n"
server_key: "d569b3ff6f96040cbe3372478c4b493e"
server_key_iv: "6d996975d21bc576ee55991e108091dc"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:43.687857 9404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:43.703629 9404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:43.707973 9404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:43.708328 9404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:43.709659 9404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:36:43.711160 9404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:36:43.711270 9404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:43.711424 9404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:36:43.711498 9404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:43.760854 9404 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:43107
I20260619 06:36:43.760854 9532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:43107 every 8 connection(s)
I20260619 06:36:43.762440 9404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:36:43.763321 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 9404
I20260619 06:36:43.763535 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:36:43.764439 29091 external_mini_cluster.cc:1428] Setting key ff4399d545bc2e269419586da6616314
Jun 19 06:36:43 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:43.806478 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:43.769951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:44729 (local address 127.28.104.254:44423)
0619 06:36:43.785637 (+ 15686us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:43.785647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:43.786972 (+ 1325us) server_negotiation.cc:407] Connection header received
0619 06:36:43.788667 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:43.788673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:43.788780 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:43.788933 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:43.790601 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.791581 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.794515 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.794978 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.795089 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:43.799782 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:43.799789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:43.799794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:43.799835 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:43.802857 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:43.803690 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:43.803695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:43.803698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:43.803849 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:43.804321 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:43.804327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:43.804330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:43.804719 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:43.804873 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:43.805892 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0619 06:36:43.806122 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15497,"thread_start_us":15342,"threads_started":1}
I20260619 06:36:43.807878 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:43.769181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:44423 (local address 127.28.104.195:44729)
0619 06:36:43.786348 (+ 17167us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:43.786417 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:43.788336 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:43.789258 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:43.789269 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:43.789765 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:43.790415 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:43.790434 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.791738 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.791745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:43.794326 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:43.794344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.795331 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.795344 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:43.795654 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:43.796623 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:43.796654 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:43.799470 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:43.803027 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:43.803038 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:43.803064 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:43.803556 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:43.803959 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:43.803965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:43.803969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:43.804152 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:43.804962 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:43.804971 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:43.805437 (+ 466us) client_negotiation.cc:769] Sending connection context
0619 06:36:43.805764 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 06:36:43.806289 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16822,"thread_start_us":16543,"threads_started":1}
I20260619 06:36:43.809651 9533 heartbeater.cc:347] Connected to a master server at 127.28.104.254:44423
I20260619 06:36:43.809983 9533 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:43.810887 9533 heartbeater.cc:511] Master 127.28.104.254:44423 requested a full tablet report, sending...
I20260619 06:36:43.812568 9038 ts_manager.cc:194] Registered new tserver with Master: 0f0d95bb5e354c86ad4a5d2dde9499f4 (127.28.104.195:43107)
I20260619 06:36:43.813750 9038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:44729
I20260619 06:36:43.819195 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:36:43.821636 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 9119
Jun 19 06:36:43 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:36:43.841351 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:43.822734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.28.104.193:45191)
0619 06:36:43.823388 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:43.823397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:43.823827 (+ 430us) server_negotiation.cc:407] Connection header received
0619 06:36:43.823957 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:43.823966 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:43.824366 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:43.824550 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:43.826433 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.828085 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.829760 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.830651 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.830804 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:43.835505 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:43.835518 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:43.835533 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:43.835572 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:43.838560 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:43.839291 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:43.839300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:43.839304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:43.839461 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:43.839852 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:43.839855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:43.839857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:43.840194 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:43.840444 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:43.840766 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 06:36:43.840988 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":296,"threads_started":1}
I20260619 06:36:43.842849 9249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:43.929141 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 9119
I20260619 06:36:43.945123 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 9260
Jun 19 06:36:43 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:36:43.963369 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:43.946169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.28.104.194:40693)
0619 06:36:43.946726 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:43.946748 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:43.946773 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:36:43.946845 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:43.946851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:43.947055 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:43.947187 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:43.948459 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.949693 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.951430 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:43.952042 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:43.952144 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:43.957203 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:43.957224 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:43.957246 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:43.957297 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:43.960155 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:43.961015 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:43.961066 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:43.961075 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:43.961236 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:43.961608 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:43.961614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:43.961617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:43.962085 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:43.962376 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:43.962733 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 06:36:43.963012 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":310,"threads_started":1}
I20260619 06:36:43.964938 9393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:44.049163 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 9260
I20260619 06:36:44.065479 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 9404
Jun 19 06:36:44 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:36:44.082491 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:44.066670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.28.104.195:43107)
0619 06:36:44.067269 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:44.067278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:44.067306 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:36:44.067366 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:44.067373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:44.067576 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:44.067786 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:44.069084 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:44.070109 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:44.071403 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:44.071974 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:44.072112 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:44.076540 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:44.076551 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:44.076566 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:44.076605 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:44.079560 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:44.080197 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:44.080204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:44.080207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:44.080357 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:44.080680 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:44.080688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:44.080691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:44.081016 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:44.081430 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:44.081876 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 06:36:44.082127 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":314,"threads_started":1}
I20260619 06:36:44.083844 9528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:44.169955 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 9404
I20260619 06:36:44.186632 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 9005
I20260619 06:36:44.188020 9069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:44.295958 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 9005
I20260619 06:36:44.319494 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:44423
--webserver_interface=127.28.104.254
--webserver_port=44221
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:44423
--ranger_config_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:43429/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.104.212:43429/kms with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:36:44.641613 9545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:44.641953 9545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:44.642119 9545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:44.649112 9545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:36:44.649262 9545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:44.649374 9545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:44.649464 9545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:36:44.649534 9545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:36:44.659279 9545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.104.212:43429/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:44423
--ranger_config_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:44423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=44221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:44.660569 9545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:44.661954 9545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:44.673785 9550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:44.673733 9553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:44.673823 9551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:44.674611 9545 server_base.cc:1034] running on GCE node
I20260619 06:36:44.675408 9545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:44.676796 9545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:44.677996 9545 hybrid_clock.cc:648] HybridClock initialized: now 1781851004677929 us; error 61 us; skew 500 ppm
I20260619 06:36:44.678345 9545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:44 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851004, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:44.681486 9545 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:36:44.682879 9545 webserver.cc:460] Webserver started at http://127.28.104.254:44221/ using document root <none> and password file <none>
I20260619 06:36:44.683522 9545 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:44.683665 9545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:36:44 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851004, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:36:44.735328 9545 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:36:44.738456 9561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:44.739845 9545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:36:44.739993 9545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "d6506013e29d48a4be04f75370fb7646"
format_stamp: "Formatted at 2026-06-19 06:36:40 on dist-test-slave-nf4n"
server_key: "a422f20397fa6e1f23641bded6c16ce2"
server_key_iv: "a6898035a418bcb521347f7dd3142499"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:44.740907 9545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:44.755944 9545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:44.759609 9545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:44.759850 9545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:44.780205 9545 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:44423
I20260619 06:36:44.780239 9612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:44423 every 8 connection(s)
I20260619 06:36:44.782299 9545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:36:44.789639 9613 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:36:44.792829 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 9545
I20260619 06:36:44.793128 9613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: Bootstrap starting.
I20260619 06:36:44.793907 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:43429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:45191
--local_ip_for_outbound_sockets=127.28.104.193
--tserver_master_addrs=127.28.104.254:44423
--webserver_port=35101
--webserver_interface=127.28.104.193
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.104.212:43429/kms with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:36:44.814638 9613 log.cc:824] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:44.823771 9613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:36:44.824316 9613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: Bootstrap complete.
I20260619 06:36:44.833653 9613 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } }
I20260619 06:36:44.834496 9613 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6506013e29d48a4be04f75370fb7646, State: Initialized, Role: FOLLOWER
I20260619 06:36:44.835114 9613 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } }
I20260619 06:36:44.835350 9613 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:36:44.835464 9613 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:36:44.835692 9613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:36:44.839552 9613 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } }
I20260619 06:36:44.840063 9613 leader_election.cc:302] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6506013e29d48a4be04f75370fb7646; no voters:
I20260619 06:36:44.840736 9613 leader_election.cc:288] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:36:44.840891 9617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:36:44.842118 9617 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [term 2 LEADER]: Becoming Leader. State: Replica: d6506013e29d48a4be04f75370fb7646, State: Running, Role: LEADER
I20260619 06:36:44.842720 9617 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } }
I20260619 06:36:44.842962 9613 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:36:44.845242 9619 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6506013e29d48a4be04f75370fb7646. Latest consensus state: current_term: 2 leader_uuid: "d6506013e29d48a4be04f75370fb7646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } } }
I20260619 06:36:44.845650 9619 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [sys.catalog]: This master's current role is: LEADER
I20260619 06:36:44.846237 9618 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d6506013e29d48a4be04f75370fb7646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6506013e29d48a4be04f75370fb7646" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44423 } } }
I20260619 06:36:44.846611 9618 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646 [sys.catalog]: This master's current role is: LEADER
I20260619 06:36:44.853824 9623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:36:44.859109 9623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:36:44.860469 9623 catalog_manager.cc:1259] Loaded cluster ID: c07d5ba05e5647e399c966cab22bce1f
I20260619 06:36:44.860618 9623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:36:44.864539 9623 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:36:44.865962 9623 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: Loaded TSK: 0
I20260619 06:36:44.867745 9623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:36:44.872991 9613 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 06:36:45.232683 9615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:45.233304 9615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:45.233517 9615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:45.245869 9615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:45.246104 9615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:45.246354 9615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:36:45.262828 9615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.104.212:43429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:45191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=35101
--tserver_master_addrs=127.28.104.254:44423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:45.264536 9615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:45.266291 9615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:45.296031 9651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:45.296494 9652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:45.301957 9615 server_base.cc:1034] running on GCE node
W20260619 06:36:45.302624 9654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:45.303524 9615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:45.304934 9615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:45.308386 9615 hybrid_clock.cc:648] HybridClock initialized: now 1781851005307807 us; error 566 us; skew 500 ppm
I20260619 06:36:45.308812 9615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:45 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851005, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:45.313402 9615 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:36:45.320405 9615 webserver.cc:460] Webserver started at http://127.28.104.193:35101/ using document root <none> and password file <none>
I20260619 06:36:45.321470 9615 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:45.321638 9615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:36:45 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851005, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:36:45.367465 9615 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s
I20260619 06:36:45.374922 9662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:45.377053 9615 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s
I20260619 06:36:45.377331 9615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "84d26332369b429e967da7b700112790"
format_stamp: "Formatted at 2026-06-19 06:36:42 on dist-test-slave-nf4n"
server_key: "495c771e2e1a980d0756ffc2d601c794"
server_key_iv: "1aa216e460ffa3b65e507247d698926a"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:45.378944 9615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:45.419924 9615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:45.425629 9615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:45.426028 9615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:45.433454 9615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:36:45.441625 9615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:36:45.441848 9615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 06:36:45.442044 9615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:36:45.442194 9615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:45.527871 9615 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:45191
I20260619 06:36:45.527887 9776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:45191 every 8 connection(s)
I20260619 06:36:45.529706 9615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:36:45.539413 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 9615
I20260619 06:36:45.540416 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:43429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:40693
--local_ip_for_outbound_sockets=127.28.104.194
--tserver_master_addrs=127.28.104.254:44423
--webserver_port=42561
--webserver_interface=127.28.104.194
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.104.212:43429/kms with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:36:45 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851005, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:45.582296 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:45.534904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:44423 (local address 127.28.104.193:34305)
0619 06:36:45.546079 (+ 11175us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:45.546147 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:45.552858 (+ 6711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:45.559832 (+ 6974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:45.559840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:45.560499 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:45.561160 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:45.561183 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:45.563708 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:45.563714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:45.565846 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:45.565863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:45.567771 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:45.567778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:45.568079 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:45.569486 (+ 1407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:45.569520 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:45.572327 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:45.576637 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:45.576648 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:45.576665 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:45.577304 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:45.577912 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:45.577917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:45.577920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:45.578206 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:45.579368 (+ 1162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:45.579376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:45.579822 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 06:36:45.580157 (+ 335us) client_negotiation.cc:240] Negotiation successful
0619 06:36:45.580767 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10748,"thread_start_us":16317,"threads_started":1}
I20260619 06:36:45.585114 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:45.534701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:34305 (local address 127.28.104.254:44423)
0619 06:36:45.557652 (+ 22951us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:45.557674 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:45.557737 (+ 63us) server_negotiation.cc:407] Connection header received
0619 06:36:45.558237 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:45.558254 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:45.558685 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:45.559133 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:45.561468 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:45.563476 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:45.566034 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:45.567123 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:45.567294 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:45.572710 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:45.572722 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:45.572745 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:45.572802 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:45.576458 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:45.577499 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:45.577509 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:45.577519 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:45.577743 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:45.578355 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:45.578361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:45.578364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:45.579074 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:45.579286 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:45.583179 (+ 3893us) server_negotiation.cc:299] Negotiation successful
0619 06:36:45.583619 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22606,"thread_start_us":26181,"threads_started":1}
I20260619 06:36:45.589012 9777 heartbeater.cc:347] Connected to a master server at 127.28.104.254:44423
I20260619 06:36:45.589398 9777 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:45.590341 9777 heartbeater.cc:511] Master 127.28.104.254:44423 requested a full tablet report, sending...
W20260619 06:36:45.593734 9777 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:44423 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:36:45.593976 9777 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:45.594281 9777 heartbeater.cc:511] Master 127.28.104.254:44423 requested a full tablet report, sending...
I20260619 06:36:45.595718 9777 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:45.596055 9777 heartbeater.cc:511] Master 127.28.104.254:44423 requested a full tablet report, sending...
W20260619 06:36:45.597348 9777 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:36:45.987596 9781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:45.988056 9781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:45.988270 9781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:45.997895 9781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:45.998059 9781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:45.998255 9781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:36:46.010097 9781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.104.212:43429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:40693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=42561
--tserver_master_addrs=127.28.104.254:44423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:46.011816 9781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:46.013564 9781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:46.046463 9795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:46.050014 9796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:46.051450 9799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:46.051693 9781 server_base.cc:1034] running on GCE node
I20260619 06:36:46.052814 9781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:46.054262 9781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:46.055577 9781 hybrid_clock.cc:648] HybridClock initialized: now 1781851006055501 us; error 81 us; skew 500 ppm
I20260619 06:36:46.055998 9781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:46 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:46.060192 9781 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:36:46.084839 9781 webserver.cc:460] Webserver started at http://127.28.104.194:42561/ using document root <none> and password file <none>
I20260619 06:36:46.085821 9781 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:46.085995 9781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:36:46 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:36:46.107965 9781 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 06:36:46.111080 9808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:46.112725 9781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:36:46.112937 9781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "2c78d0d9d6194a9ab43487e54d1dbb19"
format_stamp: "Formatted at 2026-06-19 06:36:42 on dist-test-slave-nf4n"
server_key: "ef23f91b481473e7492c0c15435b9362"
server_key_iv: "3fec6e86c7b66cdc5c1699cd19c9e3b6"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:46.114033 9781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 19 06:36:46 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:46.147401 9781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:46.153096 9781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:46.153488 9781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:46.156361 9781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:36:46.158372 9781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:36:46.158537 9781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 06:36:46.158689 9781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:36:46.158838 9781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:46.226647 9781 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:40693
I20260619 06:36:46.226691 9922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:40693 every 8 connection(s)
I20260619 06:36:46.228307 9781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:36:46.230489 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 9781
I20260619 06:36:46.231314 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.104.212:43429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:43107
--local_ip_for_outbound_sockets=127.28.104.195
--tserver_master_addrs=127.28.104.254:44423
--webserver_port=41145
--webserver_interface=127.28.104.195
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.104.212:43429/kms with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:36:46 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:46.263561 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:46.231906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:44423 (local address 127.28.104.194:40717)
0619 06:36:46.241719 (+ 9813us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:46.241781 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:46.243788 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:46.245618 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:46.245632 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:46.246326 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:46.247057 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:46.247085 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:46.248213 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:46.248218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:46.249689 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:46.249702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:46.250563 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:46.250570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:46.250808 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:46.251667 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:46.251693 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:46.254512 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:46.258719 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:46.258728 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:46.258755 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:46.259335 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:46.259808 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:46.259813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:46.259817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:46.260057 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:46.260726 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:46.260735 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:46.261225 (+ 490us) client_negotiation.cc:769] Sending connection context
0619 06:36:46.261531 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 06:36:46.262130 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9491,"thread_start_us":9219,"threads_started":1}
I20260619 06:36:46.265327 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:46.231594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:40717 (local address 127.28.104.254:44423)
0619 06:36:46.244932 (+ 13338us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:46.244936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:46.244955 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:36:46.245009 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:46.245013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:46.245146 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:46.245289 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:46.247267 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:46.248033 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:46.249862 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:46.250307 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:46.250445 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:46.254801 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:46.254809 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:46.254814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:46.254855 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:46.258535 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:46.259501 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:46.259506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:46.259510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:46.259652 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:46.260196 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:46.260201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:46.260204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:46.260566 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:46.264524 (+ 3958us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:46.264660 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 06:36:46.264916 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13224,"thread_start_us":13881,"threads_started":1}
I20260619 06:36:46.267122 9923 heartbeater.cc:347] Connected to a master server at 127.28.104.254:44423
I20260619 06:36:46.267457 9923 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:46.268325 9923 heartbeater.cc:511] Master 127.28.104.254:44423 requested a full tablet report, sending...
I20260619 06:36:46.270421 9578 ts_manager.cc:194] Registered new tserver with Master: 2c78d0d9d6194a9ab43487e54d1dbb19 (127.28.104.194:40693)
I20260619 06:36:46.272655 9578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:40717
I20260619 06:36:46.599937 9777 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:46.600373 9777 heartbeater.cc:511] Master 127.28.104.254:44423 requested a full tablet report, sending...
I20260619 06:36:46.601686 9578 ts_manager.cc:194] Registered new tserver with Master: 84d26332369b429e967da7b700112790 (127.28.104.193:45191)
I20260619 06:36:46.602808 9578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:34305
Jun 19 06:36:46 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
W20260619 06:36:46.715529 9928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:46.715927 9928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:46.716061 9928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:46.727346 9928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:46.727494 9928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:46.727661 9928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:36:46.743077 9928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.104.212:43429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:43107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=41145
--tserver_master_addrs=127.28.104.254:44423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:46.745302 9928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:46.747087 9928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:46.764456 9934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:46.771524 9933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:46.776922 9936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:46.787851 9928 server_base.cc:1034] running on GCE node
I20260619 06:36:46.788882 9928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:46.793429 9928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:46.794811 9928 hybrid_clock.cc:648] HybridClock initialized: now 1781851006794728 us; error 81 us; skew 500 ppm
I20260619 06:36:46.795209 9928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:46 dist-test-slave-nf4n krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:46.798663 9928 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:36:46.802342 9928 webserver.cc:460] Webserver started at http://127.28.104.195:41145/ using document root <none> and password file <none>
I20260619 06:36:46.803105 9928 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:46.803267 9928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:36:46 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
I20260619 06:36:46.823609 9928 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:36:46.826882 9944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:46.828510 9928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:36:46.828738 9928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4"
format_stamp: "Formatted at 2026-06-19 06:36:43 on dist-test-slave-nf4n"
server_key: "d569b3ff6f96040cbe3372478c4b493e"
server_key_iv: "6d996975d21bc576ee55991e108091dc"
server_key_version: "kuduclusterkey@0"
I20260619 06:36:46.829890 9928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:46.852391 9928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:46.857265 9928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:46.857579 9928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:46.861371 9928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:36:46.866745 9928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:36:46.866896 9928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:46.867026 9928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:36:46.867115 9928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:46.951141 9928 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:43107
I20260619 06:36:46.951174 10057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:43107 every 8 connection(s)
I20260619 06:36:46.953627 9928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:36:46.956539 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 9928
Jun 19 06:36:46 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for HTTP/127.28.104.212@KRBTEST.COM
Jun 19 06:36:47 dist-test-slave-nf4n krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:47.015533 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:46.960323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.28.104.254:44423)
0619 06:36:46.987585 (+ 27262us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:46.987595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:46.996425 (+ 8830us) server_negotiation.cc:407] Connection header received
0619 06:36:46.996474 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:46.996478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:46.996562 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:46.996667 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:46.999031 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.000173 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.002213 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.002825 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.002916 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.007042 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:47.007049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:47.007054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:47.007109 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:47.010780 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:47.011692 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:47.011698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:47.011701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:47.011851 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:47.012436 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:47.012442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:47.012445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:47.012806 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:47.013149 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.013753 (+ 604us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.014223 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2053,"server-negotiator.queue_time_us":27053,"spinlock_wait_cycles":13696,"thread_start_us":27303,"threads_started":1}
I20260619 06:36:47.021971 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:46.960368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:35511 (local address 127.28.104.254:44423)
0619 06:36:46.995435 (+ 35067us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:46.995446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:46.995479 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:36:46.996425 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:46.996430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:46.996562 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:46.996667 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:46.999294 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.001049 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.003606 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.004348 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.004477 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.013262 (+ 8785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:47.013268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:47.013273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:47.013308 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:47.017834 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:47.019154 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:47.019160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:47.019164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:47.019294 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:47.019765 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:47.019773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:47.019778 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:47.020143 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:47.020312 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.021332 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.021553 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34858,"spinlock_wait_cycles":25088,"thread_start_us":35700,"threads_started":1}
I20260619 06:36:47.023264 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:46.958872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:44423 (local address 127.28.104.195:35511)
0619 06:36:46.959884 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:46.959946 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:46.992629 (+ 32683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:46.997284 (+ 4655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:46.997300 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:46.998057 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:46.999033 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:46.999064 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.001249 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.001255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.003398 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.003413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.004790 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.004800 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.005151 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.009208 (+ 4057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:47.009242 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:47.012955 (+ 3713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:47.018038 (+ 5083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:47.018048 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:47.018068 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:47.019013 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:47.019424 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:47.019430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:47.019434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:47.019629 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:47.020306 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:47.020316 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:47.020813 (+ 497us) client_negotiation.cc:769] Sending connection context
0619 06:36:47.021190 (+ 377us) client_negotiation.cc:240] Negotiation successful
0619 06:36:47.021828 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":1278,"threads_started":1}
I20260619 06:36:47.024899 10060 heartbeater.cc:347] Connected to a master server at 127.28.104.254:44423
I20260619 06:36:47.025272 10060 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:47.026221 10060 heartbeater.cc:511] Master 127.28.104.254:44423 requested a full tablet report, sending...
I20260619 06:36:47.028059 9578 ts_manager.cc:194] Registered new tserver with Master: 0f0d95bb5e354c86ad4a5d2dde9499f4 (127.28.104.195:43107)
I20260619 06:36:47.029325 9578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:35511
I20260619 06:36:47.031767 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:36:47.058202 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.039677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.28.104.254:44423)
0619 06:36:47.039872 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:47.039879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:47.040427 (+ 548us) server_negotiation.cc:407] Connection header received
0619 06:36:47.040619 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:47.040625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:47.040725 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:47.040835 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:47.042942 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.044095 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.046336 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.047007 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.047152 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.050712 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:47.050720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:47.050725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:47.050771 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:47.054653 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:47.055582 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:47.055588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:47.055592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:47.055760 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:47.056238 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:47.056243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:47.056246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:47.056603 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:47.056857 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.057493 (+ 636us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.057800 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:36:47.065523 9578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:36:47.078420 9578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:36:47.163908 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.135984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.28.104.194:40693)
0619 06:36:47.142133 (+ 6149us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:47.142144 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:47.143328 (+ 1184us) server_negotiation.cc:407] Connection header received
0619 06:36:47.143438 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:47.143447 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:47.143729 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:47.143994 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:36:47.146111 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.147690 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.159040 (+ 11350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.161913 (+ 2873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.162069 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.162284 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.163369 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.163559 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5965,"thread_start_us":5825,"threads_started":1}
I20260619 06:36:47.163903 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.135376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40693 (local address 127.0.0.1:58212)
0619 06:36:47.142789 (+ 7413us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:47.142848 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:47.143034 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:47.145444 (+ 2410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:47.145452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:47.145487 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:36:47.145896 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.145909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.147866 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.147871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.158804 (+ 10933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.158820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.162806 (+ 3986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.162813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.163034 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.163087 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 06:36:47.163247 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 06:36:47.163478 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7062,"thread_start_us":14977,"threads_started":1}
I20260619 06:36:47.164616 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.143361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:43107 (local address 127.0.0.1:32890)
0619 06:36:47.153516 (+ 10155us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:47.153536 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:47.153671 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:47.155442 (+ 1771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:47.155451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:47.155487 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:36:47.155993 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.156008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.158643 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.158650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.160719 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.160736 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.163695 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.163703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.163878 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.163919 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:36:47.164010 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:36:47.164143 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10018,"mutex_wait_us":3,"thread_start_us":9842,"threads_started":1}
I20260619 06:36:47.166038 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.144065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.28.104.195:43107)
0619 06:36:47.154257 (+ 10192us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:47.154266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:47.154300 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:36:47.154401 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:47.154411 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:47.154790 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:47.155040 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:36:47.156261 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.158443 (+ 2182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.161561 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.163188 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.163368 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.163526 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.165310 (+ 1784us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.165564 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10007,"thread_start_us":9855,"threads_started":1}
I20260619 06:36:47.168725 9856 tablet_service.cc:1467] Processing CreateTablet for tablet 91574da22e6a40f8ac683a5bee4ddeef (DEFAULT_TABLE table=test-table [id=035304b1da25446094d24fb5d7c5dea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:36:47.173362 9993 tablet_service.cc:1467] Processing CreateTablet for tablet 91574da22e6a40f8ac683a5bee4ddeef (DEFAULT_TABLE table=test-table [id=035304b1da25446094d24fb5d7c5dea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:36:47.175096 9993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91574da22e6a40f8ac683a5bee4ddeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:47.176024 9856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91574da22e6a40f8ac683a5bee4ddeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:47.180958 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.151360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45191 (local address 127.0.0.1:36738)
0619 06:36:47.161296 (+ 9936us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:47.161318 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:47.161424 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:47.172179 (+ 10755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:47.172185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:47.172215 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:36:47.172564 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.172579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.174800 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.174805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.176861 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.176878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.180165 (+ 3287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.180172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.180364 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.180403 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:36:47.180470 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 06:36:47.180617 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9806,"thread_start_us":9608,"threads_started":1}
I20260619 06:36:47.182219 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.151713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.28.104.193:45191)
0619 06:36:47.171578 (+ 19865us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:47.171587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:47.171619 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:36:47.171718 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:47.171725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:47.171976 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:47.172207 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:36:47.173070 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.174640 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.178462 (+ 3822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.179926 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.181503 (+ 1577us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.181677 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.181754 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.181907 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19683,"thread_start_us":20680,"threads_started":1}
I20260619 06:36:47.186241 9712 tablet_service.cc:1467] Processing CreateTablet for tablet 91574da22e6a40f8ac683a5bee4ddeef (DEFAULT_TABLE table=test-table [id=035304b1da25446094d24fb5d7c5dea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:36:47.189322 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91574da22e6a40f8ac683a5bee4ddeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:47.244897 10082 tablet_bootstrap.cc:492] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: Bootstrap starting.
I20260619 06:36:47.250882 10084 tablet_bootstrap.cc:492] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: Bootstrap starting.
I20260619 06:36:47.254995 10084 tablet_bootstrap.cc:654] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:47.261188 10083 tablet_bootstrap.cc:492] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: Bootstrap starting.
I20260619 06:36:47.268623 10083 tablet_bootstrap.cc:654] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:47.271780 10084 log.cc:824] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:47.273157 10082 tablet_bootstrap.cc:654] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:47.274480 10083 log.cc:824] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:47.275825 10082 log.cc:824] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:47.277380 9923 heartbeater.cc:503] Master 127.28.104.254:44423 was elected leader, sending a full tablet report...
I20260619 06:36:47.285552 10082 tablet_bootstrap.cc:492] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: No bootstrap required, opened a new log
I20260619 06:36:47.285948 10082 ts_tablet_manager.cc:1397] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.000s
I20260619 06:36:47.286931 10084 tablet_bootstrap.cc:492] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: No bootstrap required, opened a new log
I20260619 06:36:47.287380 10084 ts_tablet_manager.cc:1397] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.000s
I20260619 06:36:47.296993 10083 tablet_bootstrap.cc:492] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: No bootstrap required, opened a new log
I20260619 06:36:47.297391 10083 ts_tablet_manager.cc:1397] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.006s
I20260619 06:36:47.298105 10082 raft_consensus.cc:348] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 } } peers { permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 } } peers { permanent_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 43107 } }
I20260619 06:36:47.298785 10082 raft_consensus.cc:374] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:47.298952 10082 raft_consensus.cc:729] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d26332369b429e967da7b700112790, State: Initialized, Role: FOLLOWER
I20260619 06:36:47.299765 10082 consensus_queue.cc:260] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 } } peers { permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 } } peers { permanent_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 43107 } }
I20260619 06:36:47.308025 10082 ts_tablet_manager.cc:1428] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: Time spent starting tablet: real 0.022s user 0.008s sys 0.010s
I20260619 06:36:47.308477 9777 heartbeater.cc:503] Master 127.28.104.254:44423 was elected leader, sending a full tablet report...
I20260619 06:36:47.318279 10084 raft_consensus.cc:348] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 } } peers { permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 } } peers { permanent_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 43107 } }
I20260619 06:36:47.318977 10084 raft_consensus.cc:374] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:47.319130 10084 raft_consensus.cc:729] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c78d0d9d6194a9ab43487e54d1dbb19, State: Initialized, Role: FOLLOWER
I20260619 06:36:47.320003 10084 consensus_queue.cc:260] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 } } peers { permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 } } peers { permanent_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 43107 } }
I20260619 06:36:47.327353 10083 raft_consensus.cc:348] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 } } peers { permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 } } peers { permanent_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 43107 } }
I20260619 06:36:47.328516 10084 ts_tablet_manager.cc:1428] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: Time spent starting tablet: real 0.041s user 0.020s sys 0.000s
I20260619 06:36:47.331753 10083 raft_consensus.cc:374] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:47.331884 10083 raft_consensus.cc:729] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0d95bb5e354c86ad4a5d2dde9499f4, State: Initialized, Role: FOLLOWER
I20260619 06:36:47.332762 10083 consensus_queue.cc:260] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 } } peers { permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 } } peers { permanent_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 43107 } }
I20260619 06:36:47.340641 10060 heartbeater.cc:503] Master 127.28.104.254:44423 was elected leader, sending a full tablet report...
I20260619 06:36:47.341513 10083 ts_tablet_manager.cc:1428] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: Time spent starting tablet: real 0.044s user 0.015s sys 0.002s
W20260619 06:36:47.480988 9924 tablet.cc:2369] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:36:47.494963 10061 tablet.cc:2369] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:36:47.501492 10090 raft_consensus.cc:482] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:36:47.501858 10090 raft_consensus.cc:504] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 } } peers { permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 } } peers { permanent_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 43107 } }
I20260619 06:36:47.503958 10090 leader_election.cc:288] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c78d0d9d6194a9ab43487e54d1dbb19 (127.28.104.194:40693), 84d26332369b429e967da7b700112790 (127.28.104.193:45191)
I20260619 06:36:47.510864 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.504306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:40693 (local address 127.28.104.195:56787)
0619 06:36:47.504917 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:47.504936 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:47.505078 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:47.505620 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:47.505626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:47.505658 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:36:47.505861 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.505872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.507076 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.507081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.508963 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.508977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.510117 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.510123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.510315 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.510342 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:36:47.510451 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 06:36:47.510581 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480}
I20260619 06:36:47.510850 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.504396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:56787 (local address 127.28.104.194:40693)
0619 06:36:47.504613 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:47.504620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:47.505248 (+ 628us) server_negotiation.cc:407] Connection header received
0619 06:36:47.505346 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:47.505353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:47.505444 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:47.505540 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:36:47.506024 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.506924 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.509163 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.509959 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.510126 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.510203 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.510477 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.510606 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:36:47.511592 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.504981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:43121 (local address 127.28.104.193:45191)
0619 06:36:47.505199 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:47.505205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:47.505788 (+ 583us) server_negotiation.cc:407] Connection header received
0619 06:36:47.505980 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:47.505985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:47.506078 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:47.506191 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:36:47.506789 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.507572 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.509302 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.510523 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.510672 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.510751 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.511234 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.511366 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:36:47.511883 9877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91574da22e6a40f8ac683a5bee4ddeef" candidate_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" is_pre_election: true
I20260619 06:36:47.512257 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.504661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45191 (local address 127.28.104.195:43121)
0619 06:36:47.505682 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:47.505698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:47.505839 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:47.506214 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:47.506218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:47.506244 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:36:47.506619 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.506631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.507726 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.507733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.509137 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:47.509160 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.510884 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.510890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:47.511059 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.511091 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:36:47.511822 (+ 731us) client_negotiation.cc:240] Negotiation successful
0619 06:36:47.511950 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":732,"threads_started":1}
I20260619 06:36:47.512523 9877 raft_consensus.cc:2454] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0d95bb5e354c86ad4a5d2dde9499f4 in term 0.
I20260619 06:36:47.513340 9945 leader_election.cc:302] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0d95bb5e354c86ad4a5d2dde9499f4, 2c78d0d9d6194a9ab43487e54d1dbb19; no voters:
I20260619 06:36:47.513394 9732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91574da22e6a40f8ac683a5bee4ddeef" candidate_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d26332369b429e967da7b700112790" is_pre_election: true
I20260619 06:36:47.513795 10090 raft_consensus.cc:2781] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:36:47.513955 10090 raft_consensus.cc:482] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:36:47.513953 9732 raft_consensus.cc:2454] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0d95bb5e354c86ad4a5d2dde9499f4 in term 0.
I20260619 06:36:47.514079 10090 raft_consensus.cc:3037] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:47.517407 10090 raft_consensus.cc:504] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 } } peers { permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 } } peers { permanent_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 43107 } }
I20260619 06:36:47.518071 10090 leader_election.cc:288] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2c78d0d9d6194a9ab43487e54d1dbb19 (127.28.104.194:40693), 84d26332369b429e967da7b700112790 (127.28.104.193:45191)
I20260619 06:36:47.518553 9732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91574da22e6a40f8ac683a5bee4ddeef" candidate_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d26332369b429e967da7b700112790"
I20260619 06:36:47.518592 9877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91574da22e6a40f8ac683a5bee4ddeef" candidate_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19"
I20260619 06:36:47.518769 9732 raft_consensus.cc:3037] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:47.518800 9877 raft_consensus.cc:3037] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:47.522171 9877 raft_consensus.cc:2454] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0d95bb5e354c86ad4a5d2dde9499f4 in term 1.
I20260619 06:36:47.522179 9732 raft_consensus.cc:2454] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0d95bb5e354c86ad4a5d2dde9499f4 in term 1.
I20260619 06:36:47.522591 9945 leader_election.cc:302] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0d95bb5e354c86ad4a5d2dde9499f4, 2c78d0d9d6194a9ab43487e54d1dbb19; no voters:
I20260619 06:36:47.522876 10090 raft_consensus.cc:2781] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:36:47.523509 10090 raft_consensus.cc:686] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0d95bb5e354c86ad4a5d2dde9499f4, State: Running, Role: LEADER
I20260619 06:36:47.524089 10090 consensus_queue.cc:237] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 } } peers { permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 } } peers { permanent_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 43107 } }
I20260619 06:36:47.529318 9578 catalog_manager.cc:5495] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f0d95bb5e354c86ad4a5d2dde9499f4 (127.28.104.195). New cstate: current_term: 1 leader_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f0d95bb5e354c86ad4a5d2dde9499f4" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 43107 } health_report { overall_health: HEALTHY } } }
W20260619 06:36:47.554809 9778 tablet.cc:2369] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:36:47.606707 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.600815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.28.104.195:43107)
0619 06:36:47.601144 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:47.601154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:47.601417 (+ 263us) server_negotiation.cc:407] Connection header received
0619 06:36:47.601686 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:47.601691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:47.601769 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:47.601927 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:36:47.602355 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.603097 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.604535 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.604959 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.605121 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.605614 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:36:47.606120 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:36:47.606293 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.606343 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.606423 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260619 06:36:47.615010 9732 raft_consensus.cc:1261] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790 [term 1 FOLLOWER]: Refusing update from remote peer 0f0d95bb5e354c86ad4a5d2dde9499f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:36:47.615012 9877 raft_consensus.cc:1261] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19 [term 1 FOLLOWER]: Refusing update from remote peer 0f0d95bb5e354c86ad4a5d2dde9499f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:36:47.616036 10099 consensus_queue.cc:1035] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d26332369b429e967da7b700112790" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:36:47.616422 10090 consensus_queue.cc:1035] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c78d0d9d6194a9ab43487e54d1dbb19" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:36:47.628973 10103 mvcc.cc:205] Tried to move back new op lower bound from 7298461727180263424 to 7298461726823567360. Current Snapshot: MvccSnapshot[applied={T|T < 7298461727180263424}]
I20260619 06:36:47.658630 9578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:36:47.658953 9578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:36:47.662917 9578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6506013e29d48a4be04f75370fb7646: Sending DeleteTablet for 3 replicas of tablet 91574da22e6a40f8ac683a5bee4ddeef
I20260619 06:36:47.664287 9856 tablet_service.cc:1514] Processing DeleteTablet for tablet 91574da22e6a40f8ac683a5bee4ddeef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:36:47 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:58212
I20260619 06:36:47.664489 9993 tablet_service.cc:1514] Processing DeleteTablet for tablet 91574da22e6a40f8ac683a5bee4ddeef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:36:47 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:32890
I20260619 06:36:47.665175 9712 tablet_service.cc:1514] Processing DeleteTablet for tablet 91574da22e6a40f8ac683a5bee4ddeef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:36:47 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:36738
I20260619 06:36:47.678508 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 9615
I20260619 06:36:47.690294 10114 tablet_replica.cc:330] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: stopping tablet replica
I20260619 06:36:47.690812 10114 raft_consensus.cc:2229] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:36:47.691375 10114 raft_consensus.cc:2258] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:36:47.692304 10113 tablet_replica.cc:330] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: stopping tablet replica
I20260619 06:36:47.692749 10113 raft_consensus.cc:2229] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:36:47.693506 10113 raft_consensus.cc:2258] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:36:47.696079 10114 ts_tablet_manager.cc:1905] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:36:47.703027 10114 ts_tablet_manager.cc:1918] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:36:47.703189 10114 log.cc:1196] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/91574da22e6a40f8ac683a5bee4ddeef
I20260619 06:36:47.703689 10114 ts_tablet_manager.cc:1939] T 91574da22e6a40f8ac683a5bee4ddeef P 84d26332369b429e967da7b700112790: Deleting consensus metadata
I20260619 06:36:47.707044 10113 ts_tablet_manager.cc:1905] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:36:47.713608 10112 tablet_replica.cc:330] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: stopping tablet replica
I20260619 06:36:47.714072 10112 raft_consensus.cc:2229] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:36:47.714577 10112 raft_consensus.cc:2258] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:36:47.716704 10112 ts_tablet_manager.cc:1905] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:36:47.717526 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.680048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.28.104.193:45191)
0619 06:36:47.680250 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:47.680257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:47.693563 (+ 13306us) server_negotiation.cc:407] Connection header received
0619 06:36:47.693806 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:47.693812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:47.693900 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:47.694019 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:47.696028 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.697215 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.699308 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.700011 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.700134 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.708365 (+ 8231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:47.708379 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:47.708397 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:47.708442 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:47.712525 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:47.713668 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:47.713770 (+ 102us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:47.713776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:47.713935 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:47.714572 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:47.714577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:47.714580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:47.715097 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:47.715436 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.716743 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.717127 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:36:47.718580 9772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:47.721545 9562 catalog_manager.cc:4841] TS 84d26332369b429e967da7b700112790 (127.28.104.193:45191): tablet 91574da22e6a40f8ac683a5bee4ddeef (table test-table [id=035304b1da25446094d24fb5d7c5dea1]) successfully deleted
I20260619 06:36:47.722270 10113 ts_tablet_manager.cc:1918] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:36:47.722460 10113 log.cc:1196] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/91574da22e6a40f8ac683a5bee4ddeef
I20260619 06:36:47.723022 10113 ts_tablet_manager.cc:1939] T 91574da22e6a40f8ac683a5bee4ddeef P 0f0d95bb5e354c86ad4a5d2dde9499f4: Deleting consensus metadata
I20260619 06:36:47.724809 9563 catalog_manager.cc:4841] TS 0f0d95bb5e354c86ad4a5d2dde9499f4 (127.28.104.195:43107): tablet 91574da22e6a40f8ac683a5bee4ddeef (table test-table [id=035304b1da25446094d24fb5d7c5dea1]) successfully deleted
I20260619 06:36:47.725622 10112 ts_tablet_manager.cc:1918] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:36:47.725795 10112 log.cc:1196] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/91574da22e6a40f8ac683a5bee4ddeef
I20260619 06:36:47.726301 10112 ts_tablet_manager.cc:1939] T 91574da22e6a40f8ac683a5bee4ddeef P 2c78d0d9d6194a9ab43487e54d1dbb19: Deleting consensus metadata
I20260619 06:36:47.727989 9562 catalog_manager.cc:4841] TS 2c78d0d9d6194a9ab43487e54d1dbb19 (127.28.104.194:40693): tablet 91574da22e6a40f8ac683a5bee4ddeef (table test-table [id=035304b1da25446094d24fb5d7c5dea1]) successfully deleted
I20260619 06:36:47.844137 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 9615
I20260619 06:36:47.861464 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 9781
I20260619 06:36:47.877935 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:47.862603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.28.104.194:40693)
0619 06:36:47.862797 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:47.862803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:47.862824 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:36:47.863125 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:47.863131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:47.863256 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:47.863359 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:47.865346 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.866058 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.867981 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:47.868382 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:47.868486 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:47.871821 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:47.871835 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:47.871851 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:47.871883 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:47.875057 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:47.875910 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:47.875917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:47.875920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:47.876059 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:47.876469 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:47.876473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:47.876475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:47.876826 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:47.877089 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:47.877448 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 06:36:47.877613 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:36:47.878639 9918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:47.980883 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 9781
I20260619 06:36:48.005061 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 9928
I20260619 06:36:48.020684 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:48.005878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.28.104.195:43107)
0619 06:36:48.006055 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:48.006060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:48.006138 (+ 78us) server_negotiation.cc:407] Connection header received
0619 06:36:48.006350 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:48.006354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:48.006419 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:48.006503 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:48.008367 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:48.009289 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:48.011257 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:48.011728 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:48.011831 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:48.014635 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:48.014647 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:48.014664 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:48.014702 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:48.017920 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:48.018599 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:48.018608 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:48.018611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:48.018750 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:48.019134 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:48.019144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:48.019147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:48.019509 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:48.019816 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:48.020118 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 06:36:48.020352 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:36:48.021353 10053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:48.142683 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 9928
I20260619 06:36:48.162526 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 9545
I20260619 06:36:48.163722 9608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:48.276219 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 9545
I20260619 06:36:48.298694 29091 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:36:48.643491 29091 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:36:48.643723 29091 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:36:48.988402 29091 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:36:48.988 UTC [8589] LOG: received smart shutdown request
2026-06-19 06:36:48.992 UTC [8589] LOG: background worker "logical replication launcher" (PID 8598) exited with exit code 1
2026-06-19 06:36:48.992 UTC [8593] LOG: shutting down
2026-06-19 06:36:49.147 UTC [8589] LOG: database system is shut down
2026-06-19T06:36:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53617 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:36:49 dist-test-slave-nf4n krb5kdc[10128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:36:49 dist-test-slave-nf4n krb5kdc[10128](info): set up 2 sockets
Jun 19 06:36:49 dist-test-slave-nf4n krb5kdc[10128](info): commencing operation
krb5kdc: starting...
W20260619 06:36:50.396960 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:36:50 dist-test-slave-nf4n krb5kdc[10128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:36:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:36:50.620503 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:37203
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:37203,127.28.104.253:34541,127.28.104.252:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:36:50.951546 10141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:50.951831 10141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:50.951967 10141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:50.959252 10141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:36:50.959362 10141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:50.959470 10141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:50.959575 10141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:36:50.959653 10141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:36:50.969400 10141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:37203,127.28.104.253:34541,127.28.104.252:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:37203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:50.970741 10141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:50.972093 10141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:50.983966 10147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:50.983871 10146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:50.984001 10149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:50.984773 10141 server_base.cc:1034] running on GCE node
I20260619 06:36:50.985687 10141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:50.987048 10141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:50.988333 10141 hybrid_clock.cc:648] HybridClock initialized: now 1781851010988274 us; error 47 us; skew 500 ppm
I20260619 06:36:50.988718 10141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:50 dist-test-slave-nf4n krb5kdc[10128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851010, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:50.991931 10141 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:36:50.993592 10141 webserver.cc:460] Webserver started at http://127.28.104.254:46335/ using document root <none> and password file <none>
I20260619 06:36:50.994302 10141 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:50.994446 10141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:50.994755 10141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:36:50.997768 10141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "e365dc6961a142079f899d7b0a71fb8b"
format_stamp: "Formatted at 2026-06-19 06:36:50 on dist-test-slave-nf4n"
server_key: "4a34637dc5e0ff38452bbe1b2ea9d32d"
server_key_iv: "6f995b2f85dd55bb6cc512ff720a3d09"
server_key_version: "encryptionkey@0"
I20260619 06:36:50.998559 10141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "e365dc6961a142079f899d7b0a71fb8b"
format_stamp: "Formatted at 2026-06-19 06:36:50 on dist-test-slave-nf4n"
server_key: "4a34637dc5e0ff38452bbe1b2ea9d32d"
server_key_iv: "6f995b2f85dd55bb6cc512ff720a3d09"
server_key_version: "encryptionkey@0"
I20260619 06:36:51.003602 10141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:36:51.007081 10156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:51.008401 10141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:36:51.008582 10141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "e365dc6961a142079f899d7b0a71fb8b"
format_stamp: "Formatted at 2026-06-19 06:36:50 on dist-test-slave-nf4n"
server_key: "4a34637dc5e0ff38452bbe1b2ea9d32d"
server_key_iv: "6f995b2f85dd55bb6cc512ff720a3d09"
server_key_version: "encryptionkey@0"
I20260619 06:36:51.008786 10141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:51.021900 10141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:51.025857 10141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:51.026139 10141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:51.044442 10141 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:37203
I20260619 06:36:51.044476 10208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:37203 every 8 connection(s)
I20260619 06:36:51.046031 10141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:36:51.049558 10209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:51.051843 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 10141
I20260619 06:36:51.052071 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:36:51.052737 29091 external_mini_cluster.cc:1428] Setting key 601e4957efcad5126f0194310483f907
I20260619 06:36:51.055887 10209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:36:51 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851010, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.104.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.253@KRBTEST.COM" created.
I20260619 06:36:51.078542 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.058981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37203 (local address 127.0.0.1:59216)
0619 06:36:51.059831 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.059875 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:51.061892 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:51.062679 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:51.062686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:51.063380 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:51.063695 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:51.063711 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.065304 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.065311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:51.066368 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:51.066379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.067281 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.067287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:51.067466 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:51.068143 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:51.068164 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:51.070604 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:51.074261 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:51.074271 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:51.074279 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:51.074821 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:51.075370 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:51.075373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:51.075376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:51.075521 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:51.076599 (+ 1078us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:51.076608 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:51.076801 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 06:36:51.077000 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 06:36:51.077433 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":386,"threads_started":1}
I20260619 06:36:51.078593 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.058981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.28.104.254:37203)
0619 06:36:51.060625 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:51.060637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:51.060684 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:36:51.062247 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:51.062252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:51.062464 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:51.062661 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:51.063899 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.065156 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.066505 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.067044 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.067238 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:51.070855 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:51.070866 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:51.070880 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:51.070919 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:51.074115 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:51.074977 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:51.074988 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:51.074998 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:51.075225 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:51.075638 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:51.075646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:51.075652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:51.076459 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:51.076700 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:51.077243 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 06:36:51.077661 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":1425,"spinlock_wait_cycles":16256,"thread_start_us":1209,"threads_started":1}
I20260619 06:36:51.081560 10209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:36:51.082736 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.082337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59358)
0619 06:36:51.082498 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.082599 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 06:36:51.083052 10158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
I20260619 06:36:51.084734 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.084273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59368)
0619 06:36:51.084476 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.084568 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:36:51.084997 10209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:34541: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab.
Entry for principal kudu/127.28.104.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab.
I20260619 06:36:51.103986 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.253:34541
--webserver_interface=127.28.104.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab
--principal=kudu/127.28.104.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:37203,127.28.104.253:34541,127.28.104.252:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:36:51.134461 10209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } attempt: 1
I20260619 06:36:51.135674 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.135110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59374)
0619 06:36:51.135339 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.135498 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 06:36:51.136785 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.136378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59378)
0619 06:36:51.136542 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.136625 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 06:36:51.136996 10209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:34541: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
I20260619 06:36:51.205284 10209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } attempt: 2
I20260619 06:36:51.206593 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.206024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59384)
0619 06:36:51.206293 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.206421 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 06:36:51.207664 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.207248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59386)
0619 06:36:51.207405 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.207506 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 06:36:51.207916 10209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:34541: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
I20260619 06:36:51.299203 10209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } attempt: 3
I20260619 06:36:51.300616 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.299965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59394)
0619 06:36:51.300276 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.300414 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260619 06:36:51.301951 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.301421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59398)
0619 06:36:51.301615 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.301739 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 06:36:51.302237 10209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:34541: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
W20260619 06:36:51.421643 10216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:51.421902 10216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:51.422040 10216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:51.428841 10216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:36:51.428929 10216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:51.429005 10216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:51.429101 10216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:36:51.429181 10216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:36:51.438606 10216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:37203,127.28.104.253:34541,127.28.104.252:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab
--principal=kudu/127.28.104.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.253:34541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.104.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:51.439837 10216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:51.441165 10216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:36:51.445520 10209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } attempt: 4
I20260619 06:36:51.446986 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.446290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59414)
0619 06:36:51.446653 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.446785 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260619 06:36:51.448186 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.447796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59430)
0619 06:36:51.447992 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.448078 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:36:51.448380 10209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:34541: Network error: Client connection negotiation failed: client connection to 127.28.104.253:34541: connect: Connection refused (error 111)
W20260619 06:36:51.453202 10222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:51.453234 10224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:51.453258 10221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:51.453843 10216 server_base.cc:1034] running on GCE node
I20260619 06:36:51.454602 10216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:51.455974 10216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:51.457185 10216 hybrid_clock.cc:648] HybridClock initialized: now 1781851011457152 us; error 57 us; skew 500 ppm
I20260619 06:36:51.457531 10216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:51 dist-test-slave-nf4n krb5kdc[10128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:51.460572 10216 init.cc:377] Logged in from keytab as kudu/127.28.104.253@KRBTEST.COM (short username kudu)
I20260619 06:36:51.462034 10216 webserver.cc:460] Webserver started at http://127.28.104.253:45795/ using document root <none> and password file <none>
I20260619 06:36:51.462658 10216 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:51.462802 10216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:51.463119 10216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:36:51.466115 10216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/data/instance:
uuid: "d1ec39db958d4f01b184810945e310d3"
format_stamp: "Formatted at 2026-06-19 06:36:51 on dist-test-slave-nf4n"
server_key: "5399e5637ed2ed5ead1210656f430662"
server_key_iv: "5870dc77421139f395c440080d959248"
server_key_version: "encryptionkey@0"
I20260619 06:36:51.466931 10216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/wal/instance:
uuid: "d1ec39db958d4f01b184810945e310d3"
format_stamp: "Formatted at 2026-06-19 06:36:51 on dist-test-slave-nf4n"
server_key: "5399e5637ed2ed5ead1210656f430662"
server_key_iv: "5870dc77421139f395c440080d959248"
server_key_version: "encryptionkey@0"
I20260619 06:36:51.472076 10216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:36:51.475495 10231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:51.476801 10216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:36:51.476974 10216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/wal
uuid: "d1ec39db958d4f01b184810945e310d3"
format_stamp: "Formatted at 2026-06-19 06:36:51 on dist-test-slave-nf4n"
server_key: "5399e5637ed2ed5ead1210656f430662"
server_key_iv: "5870dc77421139f395c440080d959248"
server_key_version: "encryptionkey@0"
I20260619 06:36:51.477205 10216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:51.489183 10216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:51.493011 10216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:51.493372 10216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:51.511569 10216 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.253:34541
I20260619 06:36:51.511587 10282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.253:34541 every 8 connection(s)
I20260619 06:36:51.513005 10216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
I20260619 06:36:51.513840 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 10216
I20260619 06:36:51.514052 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-1/wal/instance
I20260619 06:36:51.514590 29091 external_mini_cluster.cc:1428] Setting key 79b3cf4954f8c77487383a4f45692c48
I20260619 06:36:51.516758 10283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:51.524986 10283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.104.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.252@KRBTEST.COM" created.
Jun 19 06:36:51 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:51.546782 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.528147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.28.104.254:37203)
0619 06:36:51.528355 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:51.528363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:51.529531 (+ 1168us) server_negotiation.cc:407] Connection header received
0619 06:36:51.531392 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:51.531400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:51.531494 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:51.531624 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:51.533113 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.534230 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.535913 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.536370 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.536471 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:51.541000 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:51.541006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:51.541010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:51.541111 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:51.543814 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:51.544541 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:51.544548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:51.544551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:51.544671 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:51.545136 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:51.545141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:51.545145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:51.545414 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:51.545566 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:51.546242 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 06:36:51.546460 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:36:51.547695 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.528582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37203 (local address 127.0.0.1:59226)
0619 06:36:51.529353 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.529408 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:51.531129 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:51.531889 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:51.531897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:51.532369 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:51.532910 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:51.532926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.534460 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.534465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:51.535776 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:51.535789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.536699 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.536706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:51.537111 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:51.537996 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:51.538018 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:51.540749 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:51.543971 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:51.543980 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:51.543996 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:51.544419 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:51.544801 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:51.544806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:51.544810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:51.544977 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:51.545551 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:51.545559 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:51.545886 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 06:36:51.546136 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 06:36:51.546621 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":342,"threads_started":1}
I20260619 06:36:51.549472 10283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:36:51 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
I20260619 06:36:51.564960 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.550277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59442)
0619 06:36:51.550425 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.550439 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:51.550514 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:51.551570 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:51.551578 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:51.552079 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:51.552283 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:51.552296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.553655 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.553663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:51.555277 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:51.555296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.556108 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.556117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:51.556346 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:51.556992 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:51.557018 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:51.559403 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:51.562518 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:51.562524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:51.562528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:51.562883 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:51.563244 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:51.563258 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:51.563261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:51.563404 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:51.563984 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:51.563995 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:51.564126 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 06:36:51.564285 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 06:36:51.564554 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 06:36:51.565107 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.550564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.28.104.253:34541)
0619 06:36:51.551161 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:51.551167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:51.551189 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:36:51.551231 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:51.551237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:51.551401 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:51.551512 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:51.552439 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.553512 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.555452 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.555952 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.556044 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:51.559616 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:51.559626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:51.559639 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:51.559666 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:51.562364 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:51.563010 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:51.563019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:51.563022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:51.563132 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:51.563504 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:51.563512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:51.563518 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:51.563865 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:51.564070 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:51.564435 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 06:36:51.564660 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":464,"thread_start_us":343,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab.
Entry for principal kudu/127.28.104.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab.
I20260619 06:36:51.566429 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.252:46457
--webserver_interface=127.28.104.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab
--principal=kudu/127.28.104.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:37203,127.28.104.253:34541,127.28.104.252:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:36:51.567621 10283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:36:51.568827 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.568393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40334)
0619 06:36:51.568555 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.568667 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 06:36:51.569155 10232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
I20260619 06:36:51.570736 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.570397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40336)
0619 06:36:51.570552 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.570628 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:36:51.570953 10283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:46457: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
I20260619 06:36:51.620353 10283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } attempt: 1
I20260619 06:36:51.621554 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.621105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40350)
0619 06:36:51.621300 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.621411 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:36:51.622542 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.622216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40358)
0619 06:36:51.622376 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.622437 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:36:51.622753 10283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:46457: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
I20260619 06:36:51.690991 10283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } attempt: 2
I20260619 06:36:51.692131 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.691681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40364)
0619 06:36:51.691879 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.691980 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 06:36:51.693204 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.692798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40372)
0619 06:36:51.692962 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.693085 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:36:51.693415 10283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:46457: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
I20260619 06:36:51.747663 10209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } attempt: 5
Jun 19 06:36:51 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851010, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
I20260619 06:36:51.764205 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.749074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.28.104.253:34541)
0619 06:36:51.749307 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:51.749315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:51.749337 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:36:51.749409 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:51.749416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:51.749506 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:51.749634 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:51.750649 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.751803 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.754144 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.754921 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.755053 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:51.758719 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:51.758725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:51.758728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:51.758769 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:51.761578 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:51.762256 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:51.762264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:51.762269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:51.762394 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:51.762789 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:51.762794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:51.762797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:51.763087 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:51.763242 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:51.763641 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 06:36:51.763868 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:36:51.764209 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.748379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59448)
0619 06:36:51.748634 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.748658 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:51.748821 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:51.749715 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:51.749723 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:51.750211 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:51.750418 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:51.750430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.751998 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.752005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:51.753763 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:51.753783 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:51.755265 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:51.755272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:51.755488 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:51.756157 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:51.756176 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:51.758494 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:51.761747 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:51.761752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:51.761756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:51.762130 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:51.762509 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:51.762514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:51.762517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:51.762662 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:51.763207 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:51.763212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:51.763377 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 06:36:51.763519 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 06:36:51.763847 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 06:36:51.765985 10209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:36:51.767187 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.766787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40386)
0619 06:36:51.766959 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.767062 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:36:51.768289 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.767946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40388)
0619 06:36:51.768083 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.768160 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:36:51.768568 10209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:46457: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
I20260619 06:36:51.784614 10283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } attempt: 3
I20260619 06:36:51.785745 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.785282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40404)
0619 06:36:51.785500 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.785593 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 06:36:51.787032 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.786595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40412)
0619 06:36:51.786844 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.786922 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260619 06:36:51.787348 10283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:46457: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
I20260619 06:36:51.819804 10209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } attempt: 1
I20260619 06:36:51.821190 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.820623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40420)
0619 06:36:51.820834 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.820934 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 06:36:51.822266 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.821917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40426)
0619 06:36:51.822054 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.822164 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:36:51.822506 10209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:46457: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
I20260619 06:36:51.890725 10209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } attempt: 2
I20260619 06:36:51.892027 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.891491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40430)
0619 06:36:51.891778 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.891870 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 06:36:51.892985 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.892676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40432)
0619 06:36:51.892811 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.892884 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:36:51.893276 10209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:46457: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
W20260619 06:36:51.914019 10290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:51.914363 10290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:51.914535 10290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:51.922178 10290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:36:51.922314 10290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:51.922402 10290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:51.922503 10290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:36:51.922588 10290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:36:51.930580 10283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } attempt: 4
I20260619 06:36:51.931705 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.931249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40444)
0619 06:36:51.931468 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.931555 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 06:36:51.932608 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:51.932338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40458)
0619 06:36:51.932454 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:51.932518 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 06:36:51.932832 10283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:46457: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
I20260619 06:36:51.933394 10290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:37203,127.28.104.253:34541,127.28.104.252:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab
--principal=kudu/127.28.104.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.252:46457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.104.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:51.934758 10290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:51.936267 10290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:51.948334 10298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:51.948320 10295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:51.948863 10296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:51.949921 10290 server_base.cc:1034] running on GCE node
I20260619 06:36:51.950829 10290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:51.952423 10290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:51.953675 10290 hybrid_clock.cc:648] HybridClock initialized: now 1781851011953622 us; error 46 us; skew 500 ppm
I20260619 06:36:51.954047 10290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:51 dist-test-slave-nf4n krb5kdc[10128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:51.957373 10290 init.cc:377] Logged in from keytab as kudu/127.28.104.252@KRBTEST.COM (short username kudu)
I20260619 06:36:51.959018 10290 webserver.cc:460] Webserver started at http://127.28.104.252:44083/ using document root <none> and password file <none>
I20260619 06:36:51.959803 10290 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:51.959961 10290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:51.960304 10290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:36:51.963614 10290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/data/instance:
uuid: "39d11964437548808f3e134bf4511584"
format_stamp: "Formatted at 2026-06-19 06:36:51 on dist-test-slave-nf4n"
server_key: "3bf4295afcbb35c176086d12baf895e4"
server_key_iv: "b545728d811148bbe7f737d4be0796dc"
server_key_version: "encryptionkey@0"
I20260619 06:36:51.964529 10290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/wal/instance:
uuid: "39d11964437548808f3e134bf4511584"
format_stamp: "Formatted at 2026-06-19 06:36:51 on dist-test-slave-nf4n"
server_key: "3bf4295afcbb35c176086d12baf895e4"
server_key_iv: "b545728d811148bbe7f737d4be0796dc"
server_key_version: "encryptionkey@0"
I20260619 06:36:51.970383 10290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:36:51.974133 10305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:51.975646 10290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:36:51.975853 10290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/wal
uuid: "39d11964437548808f3e134bf4511584"
format_stamp: "Formatted at 2026-06-19 06:36:51 on dist-test-slave-nf4n"
server_key: "3bf4295afcbb35c176086d12baf895e4"
server_key_iv: "b545728d811148bbe7f737d4be0796dc"
server_key_version: "encryptionkey@0"
I20260619 06:36:51.976078 10290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:51.998792 10290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:51.999507 10209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } attempt: 3
I20260619 06:36:52.000705 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.000257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40464)
0619 06:36:52.000453 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:52.000540 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 06:36:52.001998 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.001495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40478)
0619 06:36:52.001765 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:52.001854 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260619 06:36:52.002233 10209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:46457: Network error: Client connection negotiation failed: client connection to 127.28.104.252:46457: connect: Connection refused (error 111)
I20260619 06:36:52.004894 10290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:52.005231 10290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:52.025525 10290 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.252:46457
I20260619 06:36:52.025552 10357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.252:46457 every 8 connection(s)
I20260619 06:36:52.027047 10290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
I20260619 06:36:52.030970 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:52.031098 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 10290
I20260619 06:36:52.031354 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850805520649-29091-0/minicluster-data/master-2/wal/instance
I20260619 06:36:52.031880 29091 external_mini_cluster.cc:1428] Setting key 11de0370d6911feb5c22473890d2bfce
I20260619 06:36:52.040830 10358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:36:52 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
Jun 19 06:36:52 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:52.059288 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.034013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.28.104.252:46457)
0619 06:36:52.034839 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.034855 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.034906 (+ 51us) server_negotiation.cc:407] Connection header received
0619 06:36:52.037677 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.037706 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.038264 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.038737 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.040156 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.042213 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.043855 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.045105 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.045278 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.050664 (+ 5386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.050675 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.050687 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.050734 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.054840 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.055765 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.055774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.055784 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.055993 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.056461 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.056464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.056466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.057087 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.057341 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.057852 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.058152 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":353,"threads_started":1}
I20260619 06:36:52.061422 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.043792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37203 (local address 127.0.0.1:59236)
0619 06:36:52.044689 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:52.044732 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:52.044957 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:52.045605 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:52.045615 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:52.046464 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:52.046845 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:52.046860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.048317 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.048325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:52.049935 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:52.049950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.051027 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.051037 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:52.051245 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.052049 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:52.052067 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:52.054739 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:52.058851 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:52.058862 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:52.058869 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:52.059332 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:52.059808 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:52.059811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:52.059817 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:52.059937 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:52.060513 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:52.060520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:52.060696 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 06:36:52.060863 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 06:36:52.061143 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":448,"threads_started":1}
I20260619 06:36:52.061692 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.043756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.28.104.254:37203)
0619 06:36:52.043984 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.043991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.044825 (+ 834us) server_negotiation.cc:407] Connection header received
0619 06:36:52.045162 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.045171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.045273 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.045351 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.047020 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.048086 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.050109 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.050805 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.050949 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.054964 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.054971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.054976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.055018 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.058695 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.059520 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.059525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.059529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.059673 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.060066 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.060076 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.060078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.060346 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.060545 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.061047 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.061294 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:36:52.063120 10358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:36:52 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
Jun 19 06:36:52 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:52.076692 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.063824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:34541 (local address 127.0.0.1:59454)
0619 06:36:52.063987 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:52.064006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:52.064093 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:52.064435 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:52.064439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:52.064771 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:52.064924 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:52.064932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.065985 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.065993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:52.066931 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:52.066942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.067710 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.067716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:52.067947 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.068473 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:52.068490 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:52.070824 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:52.074240 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:52.074247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:52.074251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:52.074625 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:52.075034 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:52.075039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:52.075043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:52.075194 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:52.075706 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:52.075710 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:52.075864 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 06:36:52.076039 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 06:36:52.076319 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 06:36:52.076711 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.063922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.28.104.253:34541)
0619 06:36:52.064120 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.064127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.064146 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:36:52.064207 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.064214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.064303 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.064411 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.065119 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.065842 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.067087 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.067552 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.067704 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.071005 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.071011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.071014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.071044 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.074087 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.074757 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.074764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.074768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.074919 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.075313 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.075324 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.075326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.075581 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.075748 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.076111 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.076332 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:36:52.077379 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.062529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.28.104.254:37203)
0619 06:36:52.063022 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.063026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.063041 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:36:52.063080 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.063083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.063136 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.063226 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.064686 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.065494 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.066693 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.067100 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.067232 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.071748 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.071755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.071761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.071804 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.074578 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.075266 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.075270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.075272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.075413 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.075795 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.075801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.075804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.076218 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.076508 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.076793 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.077066 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395}
I20260619 06:36:52.077785 10358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:36:52 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
Jun 19 06:36:52 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
I20260619 06:36:52.092701 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.078729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.28.104.252:46457)
0619 06:36:52.078925 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.078931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.078957 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:36:52.079105 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.079110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.079180 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.079289 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.080123 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.081219 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.082656 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.083263 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.083350 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.086986 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.086993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.086998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.087037 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.089961 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.090721 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.090726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.090728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.090844 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.091272 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.091277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.091279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.091649 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.091782 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.092234 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.092445 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":66}
I20260619 06:36:52.093143 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.078736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40492)
0619 06:36:52.078888 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:52.078908 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:52.079000 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:52.079292 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:52.079295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:52.079780 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:52.079972 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:52.079984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.081352 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.081359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:52.082524 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:52.082536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.083421 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.083427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:52.083639 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.084257 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:52.084287 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:52.086697 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:52.090088 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:52.090093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:52.090097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:52.090574 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:52.090963 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:52.090969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:52.090974 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:52.091137 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:52.091788 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:52.091794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:52.091965 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 06:36:52.092137 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 06:36:52.092634 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":129}
I20260619 06:36:52.095942 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.079616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.28.104.253:34541)
0619 06:36:52.080137 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.080142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.080158 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:36:52.080344 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.080351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.080421 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.080523 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.082109 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.082959 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.084708 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.085181 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.085292 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.089856 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.089861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.089864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.089896 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.093275 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.093934 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.093939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.093943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.094104 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.094536 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.094540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.094543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.094888 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.095180 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.095378 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.095599 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260619 06:36:52.098551 10358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584: Bootstrap starting.
I20260619 06:36:52.100549 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:36:52.101790 10358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:52.102815 10358 log.cc:824] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:52.105645 10358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584: No bootstrap required, opened a new log
I20260619 06:36:52.120877 10358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } }
I20260619 06:36:52.122301 10358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:52.122524 10358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d11964437548808f3e134bf4511584, State: Initialized, Role: FOLLOWER
I20260619 06:36:52.123420 10358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } }
I20260619 06:36:52.128628 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.106798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.28.104.252:46457)
0619 06:36:52.107065 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.107072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.107875 (+ 803us) server_negotiation.cc:407] Connection header received
0619 06:36:52.108188 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.108197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.108322 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.108412 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.111459 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.113091 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.115604 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.116496 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.116612 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.120563 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.120569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.120574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.120619 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.124374 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.125565 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.125571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.125574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.125735 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.126224 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.126231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.126236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.126716 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.126982 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.127971 (+ 989us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.128217 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:36:52.129666 10371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } } }
I20260619 06:36:52.130158 10371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:36:52.131479 10358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:36:52.134090 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.106322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.28.104.253:34541)
0619 06:36:52.106633 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.106643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.114623 (+ 7980us) server_negotiation.cc:407] Connection header received
0619 06:36:52.115083 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.115089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.115175 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.115247 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.117693 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.118881 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.121126 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.121832 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.121942 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.125623 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.125631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.125636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.125678 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.129558 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.131590 (+ 2032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.131596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.131599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.131735 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.132128 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.132133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.132136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.132543 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.132785 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.133351 (+ 566us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.133633 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260619 06:36:52.138577 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.106108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.28.104.254:37203)
0619 06:36:52.107286 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.107294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.121609 (+ 14315us) server_negotiation.cc:407] Connection header received
0619 06:36:52.121839 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.121845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.121934 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.122011 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.124056 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.125514 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.127975 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.128835 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.128943 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.132314 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.132320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.132325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.132373 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.135980 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.136626 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.136632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.136635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.136774 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.137168 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.137175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.137178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.137448 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.137690 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.138021 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.138257 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993}
W20260619 06:36:52.144627 10382 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:36:52.144775 10382 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:36:52.140108 10290 master_runner.cc:186] Error getting master registration for 127.28.104.254:37203: OK, instance_id { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" instance_seqno: 1781851011039233 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:36:52.149328 10290 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:36:52.179517 10209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } attempt: 4
Jun 19 06:36:52 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851010, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
I20260619 06:36:52.194087 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.180159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40506)
0619 06:36:52.180354 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:52.180376 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:52.180505 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:52.180890 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:52.180893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:52.181363 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:52.181588 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:52.181597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.183137 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.183143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:52.184593 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:52.184609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.185435 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.185441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:52.185646 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.186347 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:52.186365 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:52.188475 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:52.191575 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:52.191583 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:52.191590 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:52.191946 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:52.192365 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:52.192370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:52.192374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:52.192518 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:52.193099 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:52.193103 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:52.193264 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 06:36:52.193408 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 06:36:52.193685 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 06:36:52.195084 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.180290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.28.104.252:46457)
0619 06:36:52.180473 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.180480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.180501 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:36:52.180633 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.180639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.180755 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.180873 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.181733 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.182946 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.184756 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.185258 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.185372 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.188653 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.188662 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.188665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.188695 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.191436 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.192069 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.192081 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.192084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.192218 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.192622 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.192630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.192633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.192933 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.193117 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.193498 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.193724 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:36:52.199503 10209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b: Bootstrap starting.
I20260619 06:36:52.202782 10209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:52.203938 10209 log.cc:824] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:52.206662 10209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b: No bootstrap required, opened a new log
I20260619 06:36:52.218572 10209 raft_consensus.cc:348] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } }
I20260619 06:36:52.218992 10209 raft_consensus.cc:374] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:52.219110 10209 raft_consensus.cc:729] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e365dc6961a142079f899d7b0a71fb8b, State: Initialized, Role: FOLLOWER
I20260619 06:36:52.219781 10209 consensus_queue.cc:260] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } }
I20260619 06:36:52.221813 10385 sys_catalog.cc:455] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } } }
I20260619 06:36:52.222151 10385 sys_catalog.cc:458] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:36:52.223150 10209 sys_catalog.cc:564] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:36:52.232039 10283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } attempt: 5
W20260619 06:36:52.232648 10396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:36:52.232815 10396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 06:36:52 dist-test-slave-nf4n krb5kdc[10128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851011, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
I20260619 06:36:52.231794 10141 master_runner.cc:186] Error getting master registration for 127.28.104.253:34541: OK, instance_id { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" instance_seqno: 1781851011506223 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:36:52.245402 10141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:36:52.251739 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.232593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:46457 (local address 127.0.0.1:40518)
0619 06:36:52.233057 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:52.233078 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:52.233188 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:52.234703 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:52.234708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:52.235262 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:52.235464 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:52.235476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.238212 (+ 2736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.238218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:52.239657 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:52.239672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.240892 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.240899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:52.241145 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.241840 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:52.241864 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:52.244570 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:52.248899 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:52.248904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:52.248909 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:52.249435 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:52.249865 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:52.249872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:52.249878 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:52.250069 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:52.250716 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:52.250721 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:52.250893 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 06:36:52.251053 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 06:36:52.251346 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309}
I20260619 06:36:52.251829 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:52.232970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.28.104.252:46457)
0619 06:36:52.234235 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:52.234241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:52.234261 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:36:52.234322 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:52.234327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:52.234406 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:52.234507 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:52.235658 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.236784 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.239843 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:52.240544 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:52.240682 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:52.244792 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:52.244798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:52.244802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:52.244839 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:52.248762 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.249574 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.249580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.249584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.249738 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:52.250192 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:52.250198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:52.250206 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:52.250577 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:52.250752 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:52.251180 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 06:36:52.251429 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135}
W20260619 06:36:52.257495 10311 tablet.cc:2369] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:36:52.258119 10283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3: Bootstrap starting.
I20260619 06:36:52.262167 10283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:52.263448 10283 log.cc:824] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:52.266664 10283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3: No bootstrap required, opened a new log
I20260619 06:36:52.277741 10283 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } }
I20260619 06:36:52.278149 10283 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:52.278276 10283 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1ec39db958d4f01b184810945e310d3, State: Initialized, Role: FOLLOWER
W20260619 06:36:52.278795 10162 tablet.cc:2369] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:36:52.278932 10283 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } }
I20260619 06:36:52.280028 10399 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } } }
I20260619 06:36:52.280473 10399 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:36:52.281551 10283 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:36:52.301977 10216 master_runner.cc:263] No leader master found from master d1ec39db958d4f01b184810945e310d3
I20260619 06:36:52.302137 10216 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 06:36:52.304965 10410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:36:52.305205 10410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:36:52.425333 10385 raft_consensus.cc:482] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:36:52.425652 10385 raft_consensus.cc:504] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } }
I20260619 06:36:52.427269 10385 leader_election.cc:288] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1ec39db958d4f01b184810945e310d3 (127.28.104.253:34541), 39d11964437548808f3e134bf4511584 (127.28.104.252:46457)
I20260619 06:36:52.428254 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e365dc6961a142079f899d7b0a71fb8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1ec39db958d4f01b184810945e310d3" is_pre_election: true
I20260619 06:36:52.428850 10258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e365dc6961a142079f899d7b0a71fb8b in term 0.
I20260619 06:36:52.429848 10158 leader_election.cc:302] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1ec39db958d4f01b184810945e310d3, e365dc6961a142079f899d7b0a71fb8b; no voters:
I20260619 06:36:52.430362 10385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:36:52.430500 10385 raft_consensus.cc:482] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:36:52.430589 10385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:52.433938 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e365dc6961a142079f899d7b0a71fb8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d11964437548808f3e134bf4511584" is_pre_election: true
I20260619 06:36:52.434504 10332 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e365dc6961a142079f899d7b0a71fb8b in term 0.
I20260619 06:36:52.437698 10385 raft_consensus.cc:504] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } }
I20260619 06:36:52.440078 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e365dc6961a142079f899d7b0a71fb8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1ec39db958d4f01b184810945e310d3"
I20260619 06:36:52.440315 10258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:52.443804 10258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e365dc6961a142079f899d7b0a71fb8b in term 1.
I20260619 06:36:52.444378 10158 leader_election.cc:302] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1ec39db958d4f01b184810945e310d3, e365dc6961a142079f899d7b0a71fb8b; no voters:
I20260619 06:36:52.446027 10385 leader_election.cc:288] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [CANDIDATE]: Term 1 election: Requested vote from peers d1ec39db958d4f01b184810945e310d3 (127.28.104.253:34541), 39d11964437548808f3e134bf4511584 (127.28.104.252:46457)
I20260619 06:36:52.446705 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e365dc6961a142079f899d7b0a71fb8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d11964437548808f3e134bf4511584"
I20260619 06:36:52.446933 10332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:52.450377 10332 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e365dc6961a142079f899d7b0a71fb8b in term 1.
I20260619 06:36:52.451328 10385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:36:52.451764 10385 raft_consensus.cc:686] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [term 1 LEADER]: Becoming Leader. State: Replica: e365dc6961a142079f899d7b0a71fb8b, State: Running, Role: LEADER
I20260619 06:36:52.452606 10385 consensus_queue.cc:237] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } }
I20260619 06:36:52.466135 10385 sys_catalog.cc:455] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e365dc6961a142079f899d7b0a71fb8b. Latest consensus state: current_term: 1 leader_uuid: "e365dc6961a142079f899d7b0a71fb8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } } }
I20260619 06:36:52.466933 10385 sys_catalog.cc:458] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [sys.catalog]: This master's current role is: LEADER
I20260619 06:36:52.474308 10414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:36:52.476662 10414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 06:36:52.496300 10237 tablet.cc:2369] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:36:52.497550 10332 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [term 1 FOLLOWER]: Refusing update from remote peer e365dc6961a142079f899d7b0a71fb8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:36:52.498670 10412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:36:52.506949 10258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [term 1 FOLLOWER]: Refusing update from remote peer e365dc6961a142079f899d7b0a71fb8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:36:52.508106 10412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:36:52.539028 10399 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e365dc6961a142079f899d7b0a71fb8b. Latest consensus state: current_term: 1 leader_uuid: "e365dc6961a142079f899d7b0a71fb8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } } }
I20260619 06:36:52.539355 10399 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:36:52.545869 10415 mvcc.cc:205] Tried to move back new op lower bound from 7298461747178561536 to 7298461747018403840. Current Snapshot: MvccSnapshot[applied={T|T < 7298461747178561536}]
I20260619 06:36:52.550271 10417 mvcc.cc:205] Tried to move back new op lower bound from 7298461747178561536 to 7298461747018403840. Current Snapshot: MvccSnapshot[applied={T|T < 7298461747178561536}]
I20260619 06:36:52.558475 10371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e365dc6961a142079f899d7b0a71fb8b. Latest consensus state: current_term: 1 leader_uuid: "e365dc6961a142079f899d7b0a71fb8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } } }
I20260619 06:36:52.558820 10371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:36:52.559540 10399 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e365dc6961a142079f899d7b0a71fb8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } } }
I20260619 06:36:52.559792 10399 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:36:52.560890 10412 sys_catalog.cc:455] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e365dc6961a142079f899d7b0a71fb8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } } }
I20260619 06:36:52.561275 10412 sys_catalog.cc:458] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [sys.catalog]: This master's current role is: LEADER
I20260619 06:36:52.562523 10414 catalog_manager.cc:1347] Generated new cluster ID: cb4b8f18210d40be8f886e0aa63eedc6
I20260619 06:36:52.562620 10414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:36:52.577741 10371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e365dc6961a142079f899d7b0a71fb8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } } }
I20260619 06:36:52.578047 10371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:36:52.584472 10385 sys_catalog.cc:455] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e365dc6961a142079f899d7b0a71fb8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e365dc6961a142079f899d7b0a71fb8b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37203 } } peers { permanent_uuid: "d1ec39db958d4f01b184810945e310d3" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 34541 } } peers { permanent_uuid: "39d11964437548808f3e134bf4511584" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 46457 } } }
I20260619 06:36:52.586042 10385 sys_catalog.cc:458] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b [sys.catalog]: This master's current role is: LEADER
I20260619 06:36:52.585546 10416 mvcc.cc:205] Tried to move back new op lower bound from 7298461747178561536 to 7298461747018403840. Current Snapshot: MvccSnapshot[applied={T|T < 7298461747178561536 or (T in {7298461747178561536})}]
I20260619 06:36:52.608628 10414 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:36:52.610770 10414 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:36:52.631737 10414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e365dc6961a142079f899d7b0a71fb8b: Generated new TSK 0
I20260619 06:36:52.632992 10414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:36:53.146754 10382 catalog_manager.cc:1259] Loaded cluster ID: cb4b8f18210d40be8f886e0aa63eedc6
I20260619 06:36:53.146904 10382 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584: loading cluster ID for follower catalog manager: success
I20260619 06:36:53.149747 10382 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584: acquiring CA information for follower catalog manager: success
I20260619 06:36:53.151453 10382 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39d11964437548808f3e134bf4511584: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:36:53.307220 10410 catalog_manager.cc:1259] Loaded cluster ID: cb4b8f18210d40be8f886e0aa63eedc6
I20260619 06:36:53.307341 10410 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3: loading cluster ID for follower catalog manager: success
I20260619 06:36:53.309814 10410 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3: acquiring CA information for follower catalog manager: success
I20260619 06:36:53.311571 10410 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1ec39db958d4f01b184810945e310d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:36:53.754293 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 10141
I20260619 06:36:53.755556 10203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:53.867090 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 10141
I20260619 06:36:53.889780 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 10216
I20260619 06:36:53.890966 10278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:53.999645 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 10216
I20260619 06:36:54.019899 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 10290
I20260619 06:36:54.021114 10352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:54.129364 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 10290
2026-06-19T06:36:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4852 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:36:54 dist-test-slave-nf4n krb5kdc[10434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:36:54 dist-test-slave-nf4n krb5kdc[10434](info): set up 2 sockets
Jun 19 06:36:54 dist-test-slave-nf4n krb5kdc[10434](info): commencing operation
krb5kdc: starting...
W20260619 06:36:55.245307 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:36:55 dist-test-slave-nf4n krb5kdc[10434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:36:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:36:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:36:55.485404 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:39223
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:45835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:39223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:36:55.821281 10447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:55.821564 10447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:55.821743 10447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:55.828948 10447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:36:55.829082 10447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:55.829159 10447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:55.829250 10447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:36:55.829311 10447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:36:55.839171 10447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:45835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:39223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:39223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:55.840571 10447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:55.841925 10447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:55.854386 10452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:55.854457 10453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:55.854349 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:55.858628 10447 server_base.cc:1034] running on GCE node
I20260619 06:36:55.859400 10447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:55.860857 10447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:55.862080 10447 hybrid_clock.cc:648] HybridClock initialized: now 1781851015862037 us; error 63 us; skew 500 ppm
I20260619 06:36:55.862470 10447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:55 dist-test-slave-nf4n krb5kdc[10434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:55.865818 10447 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:36:55.867421 10447 webserver.cc:460] Webserver started at http://127.28.104.254:36823/ using document root <none> and password file <none>
I20260619 06:36:55.868098 10447 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:55.868207 10447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:55.868472 10447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:36:55.871447 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "f6a7d4dbc383437ab348775652919bdf"
format_stamp: "Formatted at 2026-06-19 06:36:55 on dist-test-slave-nf4n"
server_key: "268eddb26d70c5f5fc9b1f477a802d50"
server_key_iv: "ff8b5823211772bb6583bd04290b6118"
server_key_version: "encryptionkey@0"
I20260619 06:36:55.872129 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "f6a7d4dbc383437ab348775652919bdf"
format_stamp: "Formatted at 2026-06-19 06:36:55 on dist-test-slave-nf4n"
server_key: "268eddb26d70c5f5fc9b1f477a802d50"
server_key_iv: "ff8b5823211772bb6583bd04290b6118"
server_key_version: "encryptionkey@0"
I20260619 06:36:55.877269 10447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 06:36:55.880576 10462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:55.881944 10447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:36:55.882126 10447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "f6a7d4dbc383437ab348775652919bdf"
format_stamp: "Formatted at 2026-06-19 06:36:55 on dist-test-slave-nf4n"
server_key: "268eddb26d70c5f5fc9b1f477a802d50"
server_key_iv: "ff8b5823211772bb6583bd04290b6118"
server_key_version: "encryptionkey@0"
I20260619 06:36:55.882330 10447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:55.903429 10447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:55.908098 10447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:55.908403 10447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:55.927523 10447 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:39223
I20260619 06:36:55.927567 10514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:39223 every 8 connection(s)
I20260619 06:36:55.928916 10447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:36:55.932921 10515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:55.939344 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 10447
I20260619 06:36:55.939606 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:36:55.940541 29091 external_mini_cluster.cc:1428] Setting key 0ca4f798475aefdfd6b1356d50aa077a
I20260619 06:36:55.945494 10515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf: Bootstrap starting.
I20260619 06:36:55.949460 10515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:55.950649 10515 log.cc:824] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:55.953869 10515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf: No bootstrap required, opened a new log
Jun 19 06:36:55 dist-test-slave-nf4n krb5kdc[10434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:55.962177 10515 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a7d4dbc383437ab348775652919bdf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39223 } }
I20260619 06:36:55.962566 10515 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:55.962690 10515 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a7d4dbc383437ab348775652919bdf, State: Initialized, Role: FOLLOWER
I20260619 06:36:55.963284 10515 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a7d4dbc383437ab348775652919bdf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39223 } }
I20260619 06:36:55.963534 10515 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:36:55.963696 10515 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:36:55.963888 10515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:55.965794 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:55.942606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.28.104.254:39223)
0619 06:36:55.943787 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:55.943799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:55.943841 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:36:55.945784 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:55.945817 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:55.946233 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:55.946562 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:55.948564 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:55.950052 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:55.951812 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:55.952499 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:55.952634 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:55.957668 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:55.957675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:55.957694 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:55.957745 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:55.960959 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:55.961735 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:55.961744 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:55.961760 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:55.961920 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:55.962312 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:55.962318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:55.962321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:55.963030 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:55.963331 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:55.963832 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 06:36:55.964291 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":735,"threads_started":1}
I20260619 06:36:55.966931 10515 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a7d4dbc383437ab348775652919bdf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39223 } }
I20260619 06:36:55.967463 10515 leader_election.cc:302] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6a7d4dbc383437ab348775652919bdf; no voters:
I20260619 06:36:55.968055 10515 leader_election.cc:288] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:36:55.968228 10520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:36:55.969550 10520 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [term 1 LEADER]: Becoming Leader. State: Replica: f6a7d4dbc383437ab348775652919bdf, State: Running, Role: LEADER
I20260619 06:36:55.970222 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:36:55.970168 10520 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a7d4dbc383437ab348775652919bdf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39223 } }
I20260619 06:36:55.970443 10515 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:36:55.972801 10521 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6a7d4dbc383437ab348775652919bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a7d4dbc383437ab348775652919bdf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39223 } } }
I20260619 06:36:55.973196 10521 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [sys.catalog]: This master's current role is: LEADER
I20260619 06:36:55.974747 10526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:36:55.975255 10522 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6a7d4dbc383437ab348775652919bdf. Latest consensus state: current_term: 1 leader_uuid: "f6a7d4dbc383437ab348775652919bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a7d4dbc383437ab348775652919bdf" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39223 } } }
I20260619 06:36:55.975680 10522 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf [sys.catalog]: This master's current role is: LEADER
I20260619 06:36:55.981496 10526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:36:55.994304 10526 catalog_manager.cc:1347] Generated new cluster ID: 0b361519acd0436fa0d3c328cfd2390b
I20260619 06:36:55.994761 10526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:36:56.025148 10526 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:36:56.026983 10526 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:36:56.032593 10526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6a7d4dbc383437ab348775652919bdf: Generated new TSK 0
I20260619 06:36:56.033547 10526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:36:56.084236 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 10447
I20260619 06:36:56.085547 10510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:36:56.184383 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 10447
2026-06-19T06:36:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2050 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:36:56 dist-test-slave-nf4n krb5kdc[10543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:36:56 dist-test-slave-nf4n krb5kdc[10543](info): set up 2 sockets
Jun 19 06:36:56 dist-test-slave-nf4n krb5kdc[10543](info): commencing operation
krb5kdc: starting...
W20260619 06:36:57.305296 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:36:57 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:36:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:36:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:36:57.536943 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:45577
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:45577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:36:57.870309 10556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:57.870604 10556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:57.870697 10556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:57.877985 10556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:36:57.878098 10556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:57.878144 10556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:57.878206 10556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:36:57.878242 10556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:36:57.888190 10556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:45577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:45577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:57.889477 10556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:57.890720 10556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:57.903439 10561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:57.903532 10564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:57.903616 10562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:57.904506 10556 server_base.cc:1034] running on GCE node
I20260619 06:36:57.905390 10556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:57.906790 10556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:57.907990 10556 hybrid_clock.cc:648] HybridClock initialized: now 1781851017907926 us; error 52 us; skew 500 ppm
I20260619 06:36:57.908335 10556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:57 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:57.911695 10556 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:36:57.913221 10556 webserver.cc:460] Webserver started at http://127.28.104.254:39003/ using document root <none> and password file <none>
I20260619 06:36:57.913868 10556 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:57.913966 10556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:57.914224 10556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:36:57.917236 10556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "845fdc93e413458ba83be72fbc72fb00"
format_stamp: "Formatted at 2026-06-19 06:36:57 on dist-test-slave-nf4n"
server_key: "d619a2a36b8edcafaa482a8256d6a00d"
server_key_iv: "c3d5d16fd79ca64ad37354422d42be75"
server_key_version: "encryptionkey@0"
I20260619 06:36:57.918025 10556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "845fdc93e413458ba83be72fbc72fb00"
format_stamp: "Formatted at 2026-06-19 06:36:57 on dist-test-slave-nf4n"
server_key: "d619a2a36b8edcafaa482a8256d6a00d"
server_key_iv: "c3d5d16fd79ca64ad37354422d42be75"
server_key_version: "encryptionkey@0"
I20260619 06:36:57.923273 10556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:36:57.926765 10571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:57.928387 10556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:36:57.928666 10556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "845fdc93e413458ba83be72fbc72fb00"
format_stamp: "Formatted at 2026-06-19 06:36:57 on dist-test-slave-nf4n"
server_key: "d619a2a36b8edcafaa482a8256d6a00d"
server_key_iv: "c3d5d16fd79ca64ad37354422d42be75"
server_key_version: "encryptionkey@0"
I20260619 06:36:57.928876 10556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:57.944578 10556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:57.948848 10556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:57.949194 10556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:57.968154 10556 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:45577
I20260619 06:36:57.968189 10622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:45577 every 8 connection(s)
I20260619 06:36:57.969673 10556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:36:57.973471 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:57.979187 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 10556
I20260619 06:36:57.979415 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:36:57.980049 29091 external_mini_cluster.cc:1428] Setting key fc33888941a4f685806200a87cfc8a27
I20260619 06:36:57.985208 10623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Bootstrap starting.
I20260619 06:36:57.989053 10623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:57.990288 10623 log.cc:824] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:57.993844 10623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: No bootstrap required, opened a new log
Jun 19 06:36:57 dist-test-slave-nf4n krb5kdc[10543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:58.003974 10623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:36:58.004478 10623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:58.004606 10623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845fdc93e413458ba83be72fbc72fb00, State: Initialized, Role: FOLLOWER
I20260619 06:36:58.005352 10623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:36:58.005628 10623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:36:58.005787 10623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:36:58.005978 10623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:58.006124 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:57.982186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.28.104.254:45577)
0619 06:36:57.982994 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:57.983011 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:57.983060 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:36:57.985444 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:57.985472 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:57.985979 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:57.986401 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:57.987740 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:57.989747 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:57.991452 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:57.992551 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:57.992760 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:57.996854 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:57.996866 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:57.996892 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:57.996944 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:58.001237 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:58.001959 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:58.001968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:58.001978 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:58.002228 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:58.002620 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:58.002627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:58.002630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:58.003396 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:58.003708 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:58.004091 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 06:36:58.004587 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":353,"threads_started":1}
I20260619 06:36:58.008951 10623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:36:58.009589 10623 leader_election.cc:302] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 845fdc93e413458ba83be72fbc72fb00; no voters:
I20260619 06:36:58.010193 10623 leader_election.cc:288] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:36:58.010380 10628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:36:58.011102 10628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 1 LEADER]: Becoming Leader. State: Replica: 845fdc93e413458ba83be72fbc72fb00, State: Running, Role: LEADER
I20260619 06:36:58.011632 10628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:36:58.012254 10623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:36:58.014940 10629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "845fdc93e413458ba83be72fbc72fb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } } }
I20260619 06:36:58.015350 10629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: This master's current role is: LEADER
I20260619 06:36:58.016410 10630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 845fdc93e413458ba83be72fbc72fb00. Latest consensus state: current_term: 1 leader_uuid: "845fdc93e413458ba83be72fbc72fb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } } }
I20260619 06:36:58.016888 10630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: This master's current role is: LEADER
W20260619 06:36:58.023732 10644 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:36:58.023851 10644 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:36:58.024003 10636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:36:58.025843 10636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:36:58.034540 10636 catalog_manager.cc:1347] Generated new cluster ID: 9cb0b7929c074f9680e9710ed155eaaa
I20260619 06:36:58.034665 10636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:36:58.050639 10636 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:36:58.052541 10636 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:36:58.067730 10636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Generated new TSK 0
I20260619 06:36:58.068729 10636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:36:58.128391 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45577
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:36:58.471514 10649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:58.471875 10649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:58.472075 10649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:58.479560 10649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:58.479751 10649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:58.479967 10649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:36:58.489964 10649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:58.491379 10649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:58.492794 10649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:58.507388 10654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:58.507720 10655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:58.507429 10657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:58.510673 10649 server_base.cc:1034] running on GCE node
I20260619 06:36:58.511373 10649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:58.512436 10649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:58.513669 10649 hybrid_clock.cc:648] HybridClock initialized: now 1781851018513619 us; error 41 us; skew 500 ppm
I20260619 06:36:58.514035 10649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:58 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:58.517463 10649 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:36:58.519122 10649 webserver.cc:460] Webserver started at http://127.28.104.193:40623/ using document root <none> and password file <none>
I20260619 06:36:58.519865 10649 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:58.520018 10649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:58.520380 10649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:36:58.523435 10649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "b01bc3be45574d368cbe6efc120fd6b1"
format_stamp: "Formatted at 2026-06-19 06:36:58 on dist-test-slave-nf4n"
server_key: "2fec02a489296f3ed4ad61c8ae25d043"
server_key_iv: "1d4c54b9e5487ac4ed8c0083197c85fc"
server_key_version: "encryptionkey@0"
I20260619 06:36:58.524152 10649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "b01bc3be45574d368cbe6efc120fd6b1"
format_stamp: "Formatted at 2026-06-19 06:36:58 on dist-test-slave-nf4n"
server_key: "2fec02a489296f3ed4ad61c8ae25d043"
server_key_iv: "1d4c54b9e5487ac4ed8c0083197c85fc"
server_key_version: "encryptionkey@0"
I20260619 06:36:58.529439 10649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:36:58.532920 10664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:58.534438 10649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:36:58.534641 10649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "b01bc3be45574d368cbe6efc120fd6b1"
format_stamp: "Formatted at 2026-06-19 06:36:58 on dist-test-slave-nf4n"
server_key: "2fec02a489296f3ed4ad61c8ae25d043"
server_key_iv: "1d4c54b9e5487ac4ed8c0083197c85fc"
server_key_version: "encryptionkey@0"
I20260619 06:36:58.534893 10649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:58.546768 10649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:58.551129 10649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:58.551420 10649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:58.552423 10649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:36:58.553923 10649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:36:58.554028 10649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:58.554183 10649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:36:58.554255 10649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:58.587467 10649 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:41021
I20260619 06:36:58.587507 10778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:41021 every 8 connection(s)
I20260619 06:36:58.588843 10649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:36:58.594030 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 10649
I20260619 06:36:58.594242 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:36:58.594906 29091 external_mini_cluster.cc:1428] Setting key 05c6288ea3034514fe874be2840ffa69
Jun 19 06:36:58 dist-test-slave-nf4n krb5kdc[10543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:58.610540 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:58.591493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:51797 (local address 127.28.104.254:45577)
0619 06:36:58.591957 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:58.591962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:58.592931 (+ 969us) server_negotiation.cc:407] Connection header received
0619 06:36:58.594913 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:58.594919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:58.595034 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:58.595164 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:58.596890 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:58.597678 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:58.599295 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:58.599717 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:58.599853 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:58.603423 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:58.603432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:58.603437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:58.603486 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:58.607056 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:58.607829 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:58.607837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:58.607844 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:58.607995 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:58.608426 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:58.608433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:58.608440 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:58.608840 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:58.608992 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:58.609803 (+ 811us) server_negotiation.cc:299] Negotiation successful
0619 06:36:58.610065 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":245,"threads_started":1}
I20260619 06:36:58.611202 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:58.592070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:51797)
0619 06:36:58.592759 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:58.592799 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:58.594647 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:58.595400 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:58.595408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:58.595970 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:58.596727 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:58.596746 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:58.597844 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:58.597849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:58.599137 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:58.599153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:58.599940 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:58.599947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:58.600183 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:58.600896 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:58.600918 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:58.603185 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:58.607226 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:58.607233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:58.607249 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:58.607702 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:58.608113 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:58.608117 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:58.608120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:58.608308 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:58.608967 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:58.608973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:58.609403 (+ 430us) client_negotiation.cc:769] Sending connection context
0619 06:36:58.609656 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 06:36:58.610059 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":262,"threads_started":1}
I20260619 06:36:58.612890 10779 heartbeater.cc:347] Connected to a master server at 127.28.104.254:45577
I20260619 06:36:58.613247 10779 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:58.614215 10779 heartbeater.cc:511] Master 127.28.104.254:45577 requested a full tablet report, sending...
I20260619 06:36:58.616592 10588 ts_manager.cc:194] Registered new tserver with Master: b01bc3be45574d368cbe6efc120fd6b1 (127.28.104.193:41021)
I20260619 06:36:58.618299 10588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:51797
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:36:58.649791 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45577
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:36:58.977948 10785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:58.978282 10785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:58.978451 10785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:58.985852 10785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:58.986013 10785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:58.986202 10785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:36:58.996150 10785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:58.997579 10785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:58.998926 10785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:59.011983 10791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:59.012008 10790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:59.012183 10793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:59.013055 10785 server_base.cc:1034] running on GCE node
I20260619 06:36:59.013725 10785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:59.014631 10785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:59.015821 10785 hybrid_clock.cc:648] HybridClock initialized: now 1781851019015765 us; error 57 us; skew 500 ppm
I20260619 06:36:59.016108 10785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:59 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851019, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:59.019452 10785 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:36:59.020840 10785 webserver.cc:460] Webserver started at http://127.28.104.194:45339/ using document root <none> and password file <none>
I20260619 06:36:59.021565 10785 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:59.021672 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:59.021909 10785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:36:59.024705 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "f937264584484ea2b7e172d82ef7eb8c"
format_stamp: "Formatted at 2026-06-19 06:36:59 on dist-test-slave-nf4n"
server_key: "f9385c58d7603129722b2d8927867325"
server_key_iv: "de7000cb9aba27595635e9c0c09a7dc0"
server_key_version: "encryptionkey@0"
I20260619 06:36:59.025527 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "f937264584484ea2b7e172d82ef7eb8c"
format_stamp: "Formatted at 2026-06-19 06:36:59 on dist-test-slave-nf4n"
server_key: "f9385c58d7603129722b2d8927867325"
server_key_iv: "de7000cb9aba27595635e9c0c09a7dc0"
server_key_version: "encryptionkey@0"
I20260619 06:36:59.030433 10785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:36:59.034232 10800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:59.035511 10785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:36:59.035696 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "f937264584484ea2b7e172d82ef7eb8c"
format_stamp: "Formatted at 2026-06-19 06:36:59 on dist-test-slave-nf4n"
server_key: "f9385c58d7603129722b2d8927867325"
server_key_iv: "de7000cb9aba27595635e9c0c09a7dc0"
server_key_version: "encryptionkey@0"
I20260619 06:36:59.035890 10785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:59.056572 10785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:59.060616 10785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:59.060858 10785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:59.061764 10785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:36:59.063033 10785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:36:59.063104 10785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:59.063189 10785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:36:59.063241 10785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:59.096700 10785 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:39255
I20260619 06:36:59.096750 10914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:39255 every 8 connection(s)
I20260619 06:36:59.098049 10785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:36:59.102525 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 10785
I20260619 06:36:59.102728 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:36:59.103382 29091 external_mini_cluster.cc:1428] Setting key d3127672fd4a1b03580107a30dac590f
Jun 19 06:36:59 dist-test-slave-nf4n krb5kdc[10543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851019, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:59.122299 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.100698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:36257 (local address 127.28.104.254:45577)
0619 06:36:59.100885 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:59.100891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:59.102213 (+ 1322us) server_negotiation.cc:407] Connection header received
0619 06:36:59.104767 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:59.104773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:59.104846 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:59.104950 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:59.106990 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.107676 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.110143 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.110562 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.110652 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.114963 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:59.114972 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:59.114980 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:59.115027 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:59.118865 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:59.119595 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:59.119604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:59.119610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:59.119757 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:59.120178 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:59.120186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:59.120192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:59.120615 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:59.120780 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:59.121583 (+ 803us) server_negotiation.cc:299] Negotiation successful
0619 06:36:59.121839 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:36:59.122995 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.101292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:36257)
0619 06:36:59.102006 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:59.102083 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:59.104502 (+ 2419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:59.105254 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:59.105265 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:59.106000 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:59.106813 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.106841 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.107830 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.107838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.110010 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.110029 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.110786 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.110793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.111139 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.111923 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:59.111962 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:59.114721 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:59.119010 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:59.119017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:59.119031 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:59.119462 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:59.119864 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:59.119868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:59.119870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:59.120056 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:59.120768 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:59.120774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:59.121179 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 06:36:59.121432 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 06:36:59.121856 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":281,"threads_started":1}
I20260619 06:36:59.124547 10915 heartbeater.cc:347] Connected to a master server at 127.28.104.254:45577
I20260619 06:36:59.124876 10915 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:59.125746 10915 heartbeater.cc:511] Master 127.28.104.254:45577 requested a full tablet report, sending...
I20260619 06:36:59.127396 10588 ts_manager.cc:194] Registered new tserver with Master: f937264584484ea2b7e172d82ef7eb8c (127.28.104.194:39255)
I20260619 06:36:59.128331 10588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:36257
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:36:59.157178 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45577
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:36:59.488850 10920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:36:59.489240 10920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:36:59.489472 10920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:36:59.497406 10920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:36:59.497532 10920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:36:59.497762 10920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:36:59.508122 10920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:36:59.509563 10920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:36:59.510960 10920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:36:59.524971 10925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:59.524940 10928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:36:59.525046 10926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:36:59.526131 10920 server_base.cc:1034] running on GCE node
I20260619 06:36:59.526742 10920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:36:59.527673 10920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:36:59.528894 10920 hybrid_clock.cc:648] HybridClock initialized: now 1781851019528855 us; error 55 us; skew 500 ppm
I20260619 06:36:59.529264 10920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:36:59 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851019, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:36:59.537457 10920 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:36:59.538951 10920 webserver.cc:460] Webserver started at http://127.28.104.195:38777/ using document root <none> and password file <none>
I20260619 06:36:59.539639 10920 fs_manager.cc:359] Metadata directory not provided
I20260619 06:36:59.539739 10920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:36:59.539978 10920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:36:59.542912 10920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "7c506a1327004d3fb3e2b635887f81a0"
format_stamp: "Formatted at 2026-06-19 06:36:59 on dist-test-slave-nf4n"
server_key: "81f154cb55bfd11c2b71fdcaa6227e7b"
server_key_iv: "b9dc5507f1c810bf01d232ce4a588cad"
server_key_version: "encryptionkey@0"
I20260619 06:36:59.543627 10920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "7c506a1327004d3fb3e2b635887f81a0"
format_stamp: "Formatted at 2026-06-19 06:36:59 on dist-test-slave-nf4n"
server_key: "81f154cb55bfd11c2b71fdcaa6227e7b"
server_key_iv: "b9dc5507f1c810bf01d232ce4a588cad"
server_key_version: "encryptionkey@0"
I20260619 06:36:59.548684 10920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:36:59.552145 10935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:59.553684 10920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:36:59.553856 10920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "7c506a1327004d3fb3e2b635887f81a0"
format_stamp: "Formatted at 2026-06-19 06:36:59 on dist-test-slave-nf4n"
server_key: "81f154cb55bfd11c2b71fdcaa6227e7b"
server_key_iv: "b9dc5507f1c810bf01d232ce4a588cad"
server_key_version: "encryptionkey@0"
I20260619 06:36:59.554039 10920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:36:59.571106 10920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:36:59.575361 10920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:36:59.575654 10920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:36:59.576625 10920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:36:59.578059 10920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:36:59.578130 10920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:59.578266 10920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:36:59.578315 10920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:36:59.612036 10920 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:45871
I20260619 06:36:59.612129 11048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:45871 every 8 connection(s)
I20260619 06:36:59.613510 10920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:36:59.620645 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 10920
I20260619 06:36:59.620831 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:36:59.621495 29091 external_mini_cluster.cc:1428] Setting key abdb7ee17f95fb36015bd7e08c085451
I20260619 06:36:59.621603 10779 heartbeater.cc:503] Master 127.28.104.254:45577 was elected leader, sending a full tablet report...
Jun 19 06:36:59 dist-test-slave-nf4n krb5kdc[10543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851019, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:36:59.634133 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.616059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:35055 (local address 127.28.104.254:45577)
0619 06:36:59.616243 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:59.616247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:59.617462 (+ 1215us) server_negotiation.cc:407] Connection header received
0619 06:36:59.619347 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:59.619351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:59.619434 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:59.619516 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:59.621456 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.622546 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.624504 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.624954 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.625116 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.628461 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:59.628466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:59.628469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:59.628500 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:59.631135 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:59.631851 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:59.631855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:59.631857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:59.632017 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:59.632530 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:59.632533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:59.632535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:59.632799 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:59.632919 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:59.633716 (+ 797us) server_negotiation.cc:299] Negotiation successful
0619 06:36:59.633882 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:36:59.635140 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.616503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:35055)
0619 06:36:59.617301 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:59.617339 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:59.619080 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:59.619725 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:59.619733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:59.620410 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:36:59.621278 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.621306 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.622714 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.622720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.624367 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.624389 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.625219 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.625224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.625425 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.626042 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:36:59.626065 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:36:59.628254 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:36:59.631277 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:59.631284 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:59.631297 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:59.631730 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:59.632137 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:36:59.632143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:36:59.632148 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:36:59.632425 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:36:59.632914 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:36:59.632920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:36:59.633357 (+ 437us) client_negotiation.cc:769] Sending connection context
0619 06:36:59.633596 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 06:36:59.633982 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":280,"threads_started":1}
I20260619 06:36:59.636823 11049 heartbeater.cc:347] Connected to a master server at 127.28.104.254:45577
I20260619 06:36:59.637255 11049 heartbeater.cc:464] Registering TS with master...
I20260619 06:36:59.638034 11049 heartbeater.cc:511] Master 127.28.104.254:45577 requested a full tablet report, sending...
I20260619 06:36:59.639562 10588 ts_manager.cc:194] Registered new tserver with Master: 7c506a1327004d3fb3e2b635887f81a0 (127.28.104.195:45871)
I20260619 06:36:59.640466 10588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:35055
I20260619 06:36:59.649520 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:36:59.666982 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.654014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.28.104.254:45577)
0619 06:36:59.654215 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:59.654223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:59.654667 (+ 444us) server_negotiation.cc:407] Connection header received
0619 06:36:59.654875 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:59.654880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:59.654971 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:59.655062 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:59.656378 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.657212 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.658660 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.659099 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.659237 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.661805 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:36:59.661810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:36:59.661813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:36:59.661845 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:36:59.664235 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:59.664893 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:59.664900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:59.664904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:59.665109 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:36:59.665489 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:36:59.665494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:36:59.665497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:36:59.665829 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:36:59.666101 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:59.666396 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 06:36:59.666656 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:36:59.671355 10588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:36:59.678294 10588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:36:59.704329 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.690344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:41021 (local address 127.0.0.1:39626)
0619 06:36:59.691391 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:59.691441 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:59.691656 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:59.694002 (+ 2346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:59.694013 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:59.694055 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:36:59.694563 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.694579 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.697430 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.697439 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.700617 (+ 3178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.700638 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.703252 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.703262 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.703568 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.703625 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 06:36:59.703745 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 06:36:59.703961 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":578,"threads_started":1}
I20260619 06:36:59.704995 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.690738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.28.104.193:41021)
0619 06:36:59.693200 (+ 2462us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:59.693212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:59.693248 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:36:59.693380 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:59.693392 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:59.693710 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:59.694831 (+ 1121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:36:59.695119 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.697240 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.700913 (+ 3673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.702963 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.704224 (+ 1261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.704429 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:59.704509 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 06:36:59.704684 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2214,"thread_start_us":1995,"threads_started":1}
I20260619 06:36:59.708932 10713 tablet_service.cc:1467] Processing CreateTablet for tablet 0db237300068487eb2fcdb08ea14b7b7 (DEFAULT_TABLE table=test-table [id=fd4c7b24fcf84479aa6a8ceebad5b736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:36:59.710330 10713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db237300068487eb2fcdb08ea14b7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:59.712363 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.690366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:39255 (local address 127.0.0.1:40224)
0619 06:36:59.692316 (+ 1950us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:59.692337 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:59.692462 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:59.704484 (+ 12022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:59.704491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:59.704517 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:36:59.705039 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.705052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.707538 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.707550 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.709536 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.709551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.711447 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.711453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.711621 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.711659 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:36:59.711769 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 06:36:59.711899 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1800,"mutex_wait_us":52,"thread_start_us":1641,"threads_started":1}
I20260619 06:36:59.712363 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.690797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.28.104.194:39255)
0619 06:36:59.703833 (+ 13036us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:59.703842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:59.703870 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:36:59.703971 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:59.703979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:59.704244 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:59.705631 (+ 1387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:36:59.705869 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.707357 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.709775 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.711235 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.711401 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.711565 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:59.711881 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 06:36:59.712055 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12874,"thread_start_us":12756,"threads_started":1}
I20260619 06:36:59.712548 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.691999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:45871 (local address 127.0.0.1:42354)
0619 06:36:59.703217 (+ 11218us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:59.703239 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:59.703362 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:59.704661 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:59.704667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:59.704694 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:36:59.705432 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.705441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.707430 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.707435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.709643 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.709657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.711707 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.711711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.711898 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.711930 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:36:59.712004 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:36:59.712120 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11060,"mutex_wait_us":263,"thread_start_us":11931,"threads_started":1}
I20260619 06:36:59.713302 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.692513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.28.104.195:45871)
0619 06:36:59.704074 (+ 11561us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:59.704098 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:59.704132 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:36:59.704229 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:59.704236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:59.704456 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:59.705067 (+ 611us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:36:59.705603 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.707274 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.709814 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.711235 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.711430 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.711565 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:59.712688 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0619 06:36:59.712907 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10221,"thread_start_us":10069,"threads_started":1}
I20260619 06:36:59.716526 10983 tablet_service.cc:1467] Processing CreateTablet for tablet 0db237300068487eb2fcdb08ea14b7b7 (DEFAULT_TABLE table=test-table [id=fd4c7b24fcf84479aa6a8ceebad5b736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:36:59.716540 10849 tablet_service.cc:1467] Processing CreateTablet for tablet 0db237300068487eb2fcdb08ea14b7b7 (DEFAULT_TABLE table=test-table [id=fd4c7b24fcf84479aa6a8ceebad5b736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:36:59.717916 10983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db237300068487eb2fcdb08ea14b7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:59.717923 10849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db237300068487eb2fcdb08ea14b7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:36:59.725688 11067 tablet_bootstrap.cc:492] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: Bootstrap starting.
I20260619 06:36:59.729821 11067 tablet_bootstrap.cc:654] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:59.731001 11068 tablet_bootstrap.cc:492] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: Bootstrap starting.
I20260619 06:36:59.731914 11067 log.cc:824] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:59.733611 11069 tablet_bootstrap.cc:492] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: Bootstrap starting.
I20260619 06:36:59.735157 11068 tablet_bootstrap.cc:654] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:59.736485 11068 log.cc:824] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:59.737287 11069 tablet_bootstrap.cc:654] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: Neither blocks nor log segments found. Creating new log.
I20260619 06:36:59.737495 11067 tablet_bootstrap.cc:492] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: No bootstrap required, opened a new log
I20260619 06:36:59.737814 11067 ts_tablet_manager.cc:1397] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260619 06:36:59.738500 11069 log.cc:824] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:36:59.739737 11068 tablet_bootstrap.cc:492] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: No bootstrap required, opened a new log
I20260619 06:36:59.739993 11068 ts_tablet_manager.cc:1397] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 06:36:59.741883 11069 tablet_bootstrap.cc:492] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: No bootstrap required, opened a new log
I20260619 06:36:59.742156 11069 ts_tablet_manager.cc:1397] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 06:36:59.746784 11067 raft_consensus.cc:348] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } }
I20260619 06:36:59.747193 11067 raft_consensus.cc:374] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:59.747339 11067 raft_consensus.cc:729] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01bc3be45574d368cbe6efc120fd6b1, State: Initialized, Role: FOLLOWER
I20260619 06:36:59.748004 11067 consensus_queue.cc:260] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } }
I20260619 06:36:59.749984 11067 ts_tablet_manager.cc:1428] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 06:36:59.750916 11068 raft_consensus.cc:348] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } }
I20260619 06:36:59.751426 11068 raft_consensus.cc:374] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:59.751533 11068 raft_consensus.cc:729] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f937264584484ea2b7e172d82ef7eb8c, State: Initialized, Role: FOLLOWER
I20260619 06:36:59.752198 11068 consensus_queue.cc:260] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } }
I20260619 06:36:59.753059 11069 raft_consensus.cc:348] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } }
I20260619 06:36:59.753546 11069 raft_consensus.cc:374] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:36:59.753664 10915 heartbeater.cc:503] Master 127.28.104.254:45577 was elected leader, sending a full tablet report...
I20260619 06:36:59.753794 11069 raft_consensus.cc:729] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c506a1327004d3fb3e2b635887f81a0, State: Initialized, Role: FOLLOWER
I20260619 06:36:59.754089 11068 ts_tablet_manager.cc:1428] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260619 06:36:59.754400 11069 consensus_queue.cc:260] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } }
I20260619 06:36:59.755532 11049 heartbeater.cc:503] Master 127.28.104.254:45577 was elected leader, sending a full tablet report...
I20260619 06:36:59.756196 11069 ts_tablet_manager.cc:1428] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260619 06:36:59.841320 10780 tablet.cc:2369] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:36:59.850275 10916 tablet.cc:2369] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:36:59.865252 11050 tablet.cc:2369] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:36:59.882797 11074 raft_consensus.cc:482] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:36:59.883183 11074 raft_consensus.cc:504] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } }
I20260619 06:36:59.884953 11074 leader_election.cc:288] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01bc3be45574d368cbe6efc120fd6b1 (127.28.104.193:41021), 7c506a1327004d3fb3e2b635887f81a0 (127.28.104.195:45871)
I20260619 06:36:59.891438 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.885397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:54713 (local address 127.28.104.193:41021)
0619 06:36:59.885578 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:59.885583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:59.886115 (+ 532us) server_negotiation.cc:407] Connection header received
0619 06:36:59.886411 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:59.886432 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:59.886548 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:59.886646 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:36:59.887228 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.887946 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.889710 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.890480 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.890660 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.890742 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:59.891022 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 06:36:59.891190 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:36:59.891506 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.885393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:41021 (local address 127.28.104.194:54713)
0619 06:36:59.886007 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:59.886024 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:59.886186 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:59.886725 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:59.886730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:59.886756 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:36:59.887083 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.887093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.888076 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.888082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.889553 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.889564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.890709 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.890715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.890867 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.890891 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:36:59.890995 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:36:59.891154 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":356,"threads_started":1}
I20260619 06:36:59.891842 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.886500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:60751 (local address 127.28.104.195:45871)
0619 06:36:59.886703 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:59.886710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:59.886891 (+ 181us) server_negotiation.cc:407] Connection header received
0619 06:36:59.887110 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:59.887114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:59.887169 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:59.887262 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:36:59.887793 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.888514 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.890191 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.890942 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.891109 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.891187 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:59.891478 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 06:36:59.891602 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:36:59.891906 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.886135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:45871 (local address 127.28.104.194:60751)
0619 06:36:59.886796 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 06:36:59.886809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:36:59.886959 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:36:59.887288 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:36:59.887293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:36:59.887315 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:36:59.887628 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.887636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.888648 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.888653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.890041 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:36:59.890053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.891161 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.891166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:36:59.891329 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.891352 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:36:59.891454 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 06:36:59.891581 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":361,"threads_started":1}
I20260619 06:36:59.892489 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db237300068487eb2fcdb08ea14b7b7" candidate_uuid: "f937264584484ea2b7e172d82ef7eb8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01bc3be45574d368cbe6efc120fd6b1" is_pre_election: true
I20260619 06:36:59.892789 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db237300068487eb2fcdb08ea14b7b7" candidate_uuid: "f937264584484ea2b7e172d82ef7eb8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c506a1327004d3fb3e2b635887f81a0" is_pre_election: true
I20260619 06:36:59.892992 10734 raft_consensus.cc:2454] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f937264584484ea2b7e172d82ef7eb8c in term 0.
I20260619 06:36:59.893325 11003 raft_consensus.cc:2454] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f937264584484ea2b7e172d82ef7eb8c in term 0.
I20260619 06:36:59.893682 10802 leader_election.cc:302] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01bc3be45574d368cbe6efc120fd6b1, f937264584484ea2b7e172d82ef7eb8c; no voters:
I20260619 06:36:59.894006 11074 raft_consensus.cc:2781] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:36:59.894174 11074 raft_consensus.cc:482] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:36:59.894299 11074 raft_consensus.cc:3037] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:59.896464 11074 raft_consensus.cc:504] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } }
I20260619 06:36:59.897017 11074 leader_election.cc:288] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [CANDIDATE]: Term 1 election: Requested vote from peers b01bc3be45574d368cbe6efc120fd6b1 (127.28.104.193:41021), 7c506a1327004d3fb3e2b635887f81a0 (127.28.104.195:45871)
I20260619 06:36:59.897490 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db237300068487eb2fcdb08ea14b7b7" candidate_uuid: "f937264584484ea2b7e172d82ef7eb8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01bc3be45574d368cbe6efc120fd6b1"
I20260619 06:36:59.897550 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db237300068487eb2fcdb08ea14b7b7" candidate_uuid: "f937264584484ea2b7e172d82ef7eb8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c506a1327004d3fb3e2b635887f81a0"
I20260619 06:36:59.897710 10734 raft_consensus.cc:3037] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:59.897733 11003 raft_consensus.cc:3037] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:36:59.899822 11003 raft_consensus.cc:2454] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f937264584484ea2b7e172d82ef7eb8c in term 1.
I20260619 06:36:59.899822 10734 raft_consensus.cc:2454] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f937264584484ea2b7e172d82ef7eb8c in term 1.
I20260619 06:36:59.900229 10802 leader_election.cc:302] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01bc3be45574d368cbe6efc120fd6b1, f937264584484ea2b7e172d82ef7eb8c; no voters:
I20260619 06:36:59.900489 11074 raft_consensus.cc:2781] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:36:59.900944 11074 raft_consensus.cc:686] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 1 LEADER]: Becoming Leader. State: Replica: f937264584484ea2b7e172d82ef7eb8c, State: Running, Role: LEADER
I20260619 06:36:59.901489 11074 consensus_queue.cc:237] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } }
I20260619 06:36:59.906709 10588 catalog_manager.cc:5495] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c reported cstate change: term changed from 0 to 1, leader changed from <none> to f937264584484ea2b7e172d82ef7eb8c (127.28.104.194). New cstate: current_term: 1 leader_uuid: "f937264584484ea2b7e172d82ef7eb8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } health_report { overall_health: UNKNOWN } } }
I20260619 06:36:59.942495 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.936945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.28.104.194:39255)
0619 06:36:59.937184 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:59.937193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:59.937227 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:36:59.937325 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:59.937329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:59.937415 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:59.937565 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:36:59.937982 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.938717 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.940246 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.940673 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:36:59.940808 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:36:59.941379 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:36:59.941897 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:36:59.942074 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 06:36:59.942125 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 06:36:59.942206 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:36:59.950578 10734 raft_consensus.cc:1261] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1 [term 1 FOLLOWER]: Refusing update from remote peer f937264584484ea2b7e172d82ef7eb8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:36:59.950590 11003 raft_consensus.cc:1261] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0 [term 1 FOLLOWER]: Refusing update from remote peer f937264584484ea2b7e172d82ef7eb8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:36:59.951566 11078 consensus_queue.cc:1035] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:36:59.952051 11074 consensus_queue.cc:1035] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:36:59.962847 11081 mvcc.cc:205] Tried to move back new op lower bound from 7298461777707888640 to 7298461777520066560. Current Snapshot: MvccSnapshot[applied={T|T < 7298461777707888640}]
I20260619 06:36:59.967679 11083 mvcc.cc:205] Tried to move back new op lower bound from 7298461777707888640 to 7298461777520066560. Current Snapshot: MvccSnapshot[applied={T|T < 7298461777707888640 or (T in {7298461777707888640})}]
I20260619 06:36:59.969441 11082 mvcc.cc:205] Tried to move back new op lower bound from 7298461777707888640 to 7298461777520066560. Current Snapshot: MvccSnapshot[applied={T|T < 7298461777707888640 or (T in {7298461777707888640})}]
I20260619 06:36:59.977092 10588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:36:59.977355 10588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:36:59.981444 10588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Sending DeleteTablet for 3 replicas of tablet 0db237300068487eb2fcdb08ea14b7b7
I20260619 06:36:59.982589 10849 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db237300068487eb2fcdb08ea14b7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:36:59 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:40224
I20260619 06:36:59.982595 10713 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db237300068487eb2fcdb08ea14b7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:36:59 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:39626
I20260619 06:36:59.984789 10983 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db237300068487eb2fcdb08ea14b7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:36:59 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:42354
I20260619 06:36:59.984933 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 10649
I20260619 06:36:59.997171 11090 tablet_replica.cc:330] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: stopping tablet replica
I20260619 06:36:59.997679 11091 tablet_replica.cc:330] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: stopping tablet replica
I20260619 06:36:59.997764 11090 raft_consensus.cc:2229] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:36:59.998129 11091 raft_consensus.cc:2229] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:36:59.998454 11090 raft_consensus.cc:2258] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:36:59.998680 11091 raft_consensus.cc:2258] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:36:59.999323 11092 tablet_replica.cc:330] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: stopping tablet replica
I20260619 06:36:59.999743 11092 raft_consensus.cc:2229] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:37:00.000260 11092 raft_consensus.cc:2258] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:37:00.001348 11091 ts_tablet_manager.cc:1905] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:00.002231 11090 ts_tablet_manager.cc:1905] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:00.002470 11092 ts_tablet_manager.cc:1905] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:00.009882 11090 ts_tablet_manager.cc:1918] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:00.010186 11090 log.cc:1196] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/0db237300068487eb2fcdb08ea14b7b7
I20260619 06:37:00.010905 11090 ts_tablet_manager.cc:1939] T 0db237300068487eb2fcdb08ea14b7b7 P f937264584484ea2b7e172d82ef7eb8c: Deleting consensus metadata
I20260619 06:37:00.013077 10575 catalog_manager.cc:4841] TS f937264584484ea2b7e172d82ef7eb8c (127.28.104.194:39255): tablet 0db237300068487eb2fcdb08ea14b7b7 (table test-table [id=fd4c7b24fcf84479aa6a8ceebad5b736]) successfully deleted
Jun 19 06:37:00 dist-test-slave-nf4n krb5kdc[10543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:37:00.023149 11091 ts_tablet_manager.cc:1918] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:00.023519 11091 log.cc:1196] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/0db237300068487eb2fcdb08ea14b7b7
I20260619 06:37:00.023793 11092 ts_tablet_manager.cc:1918] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:00.023980 11092 log.cc:1196] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/0db237300068487eb2fcdb08ea14b7b7
I20260619 06:37:00.024384 11091 ts_tablet_manager.cc:1939] T 0db237300068487eb2fcdb08ea14b7b7 P b01bc3be45574d368cbe6efc120fd6b1: Deleting consensus metadata
I20260619 06:37:00.024537 11092 ts_tablet_manager.cc:1939] T 0db237300068487eb2fcdb08ea14b7b7 P 7c506a1327004d3fb3e2b635887f81a0: Deleting consensus metadata
I20260619 06:37:00.026585 10572 catalog_manager.cc:4841] TS b01bc3be45574d368cbe6efc120fd6b1 (127.28.104.193:41021): tablet 0db237300068487eb2fcdb08ea14b7b7 (table test-table [id=fd4c7b24fcf84479aa6a8ceebad5b736]) successfully deleted
I20260619 06:37:00.026924 10572 catalog_manager.cc:4841] TS 7c506a1327004d3fb3e2b635887f81a0 (127.28.104.195:45871): tablet 0db237300068487eb2fcdb08ea14b7b7 (table test-table [id=fd4c7b24fcf84479aa6a8ceebad5b736]) successfully deleted
I20260619 06:37:00.027808 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:36:59.985710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.28.104.193:41021)
0619 06:36:59.985882 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:36:59.985888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:36:59.989581 (+ 3693us) server_negotiation.cc:407] Connection header received
0619 06:36:59.990327 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:36:59.990337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:36:59.990457 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:36:59.990550 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:36:59.993159 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:36:59.994689 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:00.001583 (+ 6894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:00.002215 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:00.002360 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:00.019512 (+ 17152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:00.019529 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:00.019549 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:00.019595 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:00.024000 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:00.025101 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:00.025110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:00.025114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:00.025298 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:00.025779 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:00.025785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:00.025788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:00.026274 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:00.026564 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:00.027193 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 06:37:00.027464 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:37:00.029378 10774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:00.146360 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 10649
I20260619 06:37:00.170274 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 10785
Jun 19 06:37:00 dist-test-slave-nf4n krb5kdc[10543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:37:00.194418 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:00.171149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.28.104.194:39255)
0619 06:37:00.171341 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:00.171346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:00.171383 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:37:00.171635 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:00.171641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:00.171762 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:00.171862 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:00.173650 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:00.174541 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:00.176454 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:00.176884 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:00.176993 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:00.187732 (+ 10739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:00.187749 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:00.187765 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:00.187810 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:00.191172 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:00.192060 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:00.192068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:00.192070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:00.192272 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:00.192714 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:00.192718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:00.192720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:00.193240 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:00.193486 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:00.193851 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 06:37:00.194019 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:37:00.195223 10910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:00.311470 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 10785
I20260619 06:37:00.330310 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 10920
Jun 19 06:37:00 dist-test-slave-nf4n krb5kdc[10543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:37:00.345602 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:00.331186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.28.104.195:45871)
0619 06:37:00.331361 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:00.331367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:00.331390 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:37:00.331467 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:00.331472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:00.331531 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:00.331636 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:00.332919 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:00.333757 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:00.335017 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:00.335413 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:00.335551 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:00.339974 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:00.339990 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:00.340006 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:00.340048 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:00.342808 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:00.343496 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:00.343503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:00.343505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:00.343652 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:00.344029 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:00.344035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:00.344038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:00.344380 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:00.344699 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:00.344990 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 06:37:00.345247 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:37:00.346276 11044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:00.434765 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 10920
I20260619 06:37:00.452271 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 10556
I20260619 06:37:00.453531 10618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:00.565207 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 10556
I20260619 06:37:00.586918 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:45577
--webserver_interface=127.28.104.254
--webserver_port=39003
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:45577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:37:00.913637 11098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:00.913926 11098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:00.914090 11098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:00.922544 11098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:37:00.922644 11098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:00.922744 11098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:00.922852 11098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:37:00.922938 11098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:37:00.932762 11098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:45577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:45577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=39003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:00.934132 11098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:00.935442 11098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:00.958148 11105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:00.958333 11104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:00.958106 11107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:00.959520 11098 server_base.cc:1034] running on GCE node
I20260619 06:37:00.960456 11098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:00.962100 11098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:00.963362 11098 hybrid_clock.cc:648] HybridClock initialized: now 1781851020963307 us; error 36 us; skew 500 ppm
I20260619 06:37:00.963788 11098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:00 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:00.967357 11098 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:37:00.969260 11098 webserver.cc:460] Webserver started at http://127.28.104.254:39003/ using document root <none> and password file <none>
I20260619 06:37:00.970037 11098 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:00.970206 11098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:00.976272 11098 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 06:37:00.979468 11114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:00.980872 11098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:37:00.981123 11098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "845fdc93e413458ba83be72fbc72fb00"
format_stamp: "Formatted at 2026-06-19 06:36:57 on dist-test-slave-nf4n"
server_key: "d619a2a36b8edcafaa482a8256d6a00d"
server_key_iv: "c3d5d16fd79ca64ad37354422d42be75"
server_key_version: "encryptionkey@0"
I20260619 06:37:00.982208 11098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:00.995524 11098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:00.999974 11098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:01.000272 11098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:01.019886 11098 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:45577
I20260619 06:37:01.019908 11165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:45577 every 8 connection(s)
I20260619 06:37:01.021574 11098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:37:01.028644 11166 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:37:01.029698 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 11098
I20260619 06:37:01.030684 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:41021
--local_ip_for_outbound_sockets=127.28.104.193
--tserver_master_addrs=127.28.104.254:45577
--webserver_port=40623
--webserver_interface=127.28.104.193
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:37:01.033794 11166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Bootstrap starting.
I20260619 06:37:01.055382 11166 log.cc:824] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:01.067783 11166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:37:01.068415 11166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Bootstrap complete.
I20260619 06:37:01.077677 11166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:37:01.078583 11166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845fdc93e413458ba83be72fbc72fb00, State: Initialized, Role: FOLLOWER
I20260619 06:37:01.079222 11166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:37:01.079461 11166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:37:01.079576 11166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:37:01.079762 11166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:37:01.083676 11166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:37:01.084244 11166 leader_election.cc:302] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 845fdc93e413458ba83be72fbc72fb00; no voters:
I20260619 06:37:01.084892 11166 leader_election.cc:288] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:37:01.085047 11170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:37:01.086434 11170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 2 LEADER]: Becoming Leader. State: Replica: 845fdc93e413458ba83be72fbc72fb00, State: Running, Role: LEADER
I20260619 06:37:01.087075 11170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:37:01.087370 11166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:37:01.090015 11171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "845fdc93e413458ba83be72fbc72fb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } } }
I20260619 06:37:01.090304 11171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: This master's current role is: LEADER
I20260619 06:37:01.089567 11172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 845fdc93e413458ba83be72fbc72fb00. Latest consensus state: current_term: 2 leader_uuid: "845fdc93e413458ba83be72fbc72fb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } } }
I20260619 06:37:01.090869 11172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: This master's current role is: LEADER
I20260619 06:37:01.095803 11177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:37:01.105973 11177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:37:01.108474 11177 catalog_manager.cc:1259] Loaded cluster ID: 9cb0b7929c074f9680e9710ed155eaaa
I20260619 06:37:01.108815 11177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:37:01.113359 11177 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 06:37:01.140151 11177 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 06:37:01.144021 11188 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:37:01.148914 11177 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:37:01.150409 11177 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Loaded TSK: 0
I20260619 06:37:01.152046 11177 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:37:01.398350 11168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:01.398681 11168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:01.398856 11168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:01.406278 11168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:01.406433 11168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:01.406601 11168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:37:01.416724 11168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:41021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=40623
--tserver_master_addrs=127.28.104.254:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:01.418072 11168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:01.419431 11168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:01.434677 11195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:01.435151 11194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:01.435632 11197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:01.435812 11168 server_base.cc:1034] running on GCE node
I20260619 06:37:01.436491 11168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:01.437579 11168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:01.438832 11168 hybrid_clock.cc:648] HybridClock initialized: now 1781851021438773 us; error 55 us; skew 500 ppm
I20260619 06:37:01.439162 11168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:01 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:01.442366 11168 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:37:01.443956 11168 webserver.cc:460] Webserver started at http://127.28.104.193:40623/ using document root <none> and password file <none>
I20260619 06:37:01.444646 11168 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:01.444792 11168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:01.450155 11168 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 06:37:01.452986 11204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:01.454543 11168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:37:01.454711 11168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "b01bc3be45574d368cbe6efc120fd6b1"
format_stamp: "Formatted at 2026-06-19 06:36:58 on dist-test-slave-nf4n"
server_key: "2fec02a489296f3ed4ad61c8ae25d043"
server_key_iv: "1d4c54b9e5487ac4ed8c0083197c85fc"
server_key_version: "encryptionkey@0"
I20260619 06:37:01.455760 11168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:01.486582 11168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:01.491285 11168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:01.491577 11168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:01.492587 11168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:01.494007 11168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:01.494102 11168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:01.494261 11168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:01.494339 11168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:01.528383 11168 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:41021
I20260619 06:37:01.528484 11316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:41021 every 8 connection(s)
I20260619 06:37:01.529893 11168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:37:01.537357 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 11168
I20260619 06:37:01.537981 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:39255
--local_ip_for_outbound_sockets=127.28.104.194
--tserver_master_addrs=127.28.104.254:45577
--webserver_port=45339
--webserver_interface=127.28.104.194
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:37:01 dist-test-slave-nf4n krb5kdc[10543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:37:01.553467 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:01.532730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:48741 (local address 127.28.104.254:45577)
0619 06:37:01.533461 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:01.533479 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:01.533875 (+ 396us) server_negotiation.cc:407] Connection header received
0619 06:37:01.535859 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:01.535871 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:01.536183 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:01.536467 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:01.537999 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:01.539299 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:01.540841 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:01.541557 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:01.541772 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:01.545683 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:01.545699 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:01.545714 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:01.545755 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:01.548623 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:01.549601 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:01.549607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:01.549613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:01.549780 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:01.550274 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:01.550278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:01.550280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:01.550833 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:01.550977 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:01.552003 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0619 06:37:01.552420 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":330,"threads_started":1}
I20260619 06:37:01.553763 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:01.532890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:48741)
0619 06:37:01.533703 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:01.533746 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:01.535378 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:01.536609 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:01.536616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:01.537184 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:37:01.537770 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:01.537787 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:01.539463 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:01.539473 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:01.540685 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:01.540708 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:01.541799 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:01.541807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:01.542093 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:01.542785 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:37:01.542807 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:37:01.545418 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:37:01.548766 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:01.548777 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:01.548796 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:01.549469 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:01.549899 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:01.549907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:01.549914 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:01.550163 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:01.550970 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:37:01.550980 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:37:01.551449 (+ 469us) client_negotiation.cc:769] Sending connection context
0619 06:37:01.551772 (+ 323us) client_negotiation.cc:240] Negotiation successful
0619 06:37:01.552348 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":329,"threads_started":1}
I20260619 06:37:01.556826 11317 heartbeater.cc:347] Connected to a master server at 127.28.104.254:45577
I20260619 06:37:01.557238 11317 heartbeater.cc:464] Registering TS with master...
I20260619 06:37:01.558224 11317 heartbeater.cc:511] Master 127.28.104.254:45577 requested a full tablet report, sending...
I20260619 06:37:01.560779 11131 ts_manager.cc:194] Registered new tserver with Master: b01bc3be45574d368cbe6efc120fd6b1 (127.28.104.193:41021)
I20260619 06:37:01.562256 11131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:48741
W20260619 06:37:01.870271 11321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:01.870544 11321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:01.870702 11321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:01.877990 11321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:01.878134 11321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:01.878325 11321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:37:01.888195 11321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:39255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=45339
--tserver_master_addrs=127.28.104.254:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:01.889592 11321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:01.890928 11321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:01.903924 11326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:01.904045 11327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:01.904114 11329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:01.905172 11321 server_base.cc:1034] running on GCE node
I20260619 06:37:01.905758 11321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:01.906731 11321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:01.907965 11321 hybrid_clock.cc:648] HybridClock initialized: now 1781851021907905 us; error 56 us; skew 500 ppm
I20260619 06:37:01.908313 11321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:01 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:01.911640 11321 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:37:01.913125 11321 webserver.cc:460] Webserver started at http://127.28.104.194:45339/ using document root <none> and password file <none>
I20260619 06:37:01.913796 11321 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:01.913936 11321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:01.919760 11321 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:37:01.922937 11337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:01.924261 11321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:37:01.924429 11321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "f937264584484ea2b7e172d82ef7eb8c"
format_stamp: "Formatted at 2026-06-19 06:36:59 on dist-test-slave-nf4n"
server_key: "f9385c58d7603129722b2d8927867325"
server_key_iv: "de7000cb9aba27595635e9c0c09a7dc0"
server_key_version: "encryptionkey@0"
I20260619 06:37:01.925472 11321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:01.949345 11321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:01.954159 11321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:01.954438 11321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:01.955443 11321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:01.956813 11321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:01.956902 11321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:01.957075 11321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:01.957170 11321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 06:37:01.990155 11321 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:39255
I20260619 06:37:01.990248 11449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:39255 every 8 connection(s)
I20260619 06:37:01.991647 11321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:37:01.993059 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 11321
I20260619 06:37:01.993788 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:45871
--local_ip_for_outbound_sockets=127.28.104.195
--tserver_master_addrs=127.28.104.254:45577
--webserver_port=38777
--webserver_interface=127.28.104.195
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:37:02 dist-test-slave-nf4n krb5kdc[10543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:37:02.012861 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:01.994621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:42925 (local address 127.28.104.254:45577)
0619 06:37:01.994809 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:01.994814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:01.996254 (+ 1440us) server_negotiation.cc:407] Connection header received
0619 06:37:01.998244 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:01.998252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:01.998349 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:01.998460 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:01.999981 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.000694 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.002332 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.002772 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.002908 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.006936 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:02.006941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:02.006945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:02.006978 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:02.009681 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:02.010545 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:02.010550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:02.010552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:02.010670 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:02.011117 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:02.011129 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:02.011131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:02.011399 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:02.011554 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.012261 (+ 707us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.012491 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:37:02.013880 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:01.995329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:42925)
0619 06:37:01.996079 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:01.996126 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:01.997959 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:01.998716 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:01.998725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:01.999241 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:37:01.999803 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:01.999820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.000854 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.000859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.002171 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.002193 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.003003 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.003011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.003238 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.003929 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:37:02.003952 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:37:02.006691 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:37:02.009846 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:02.009856 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:02.009873 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:02.010406 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:02.010794 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:02.010799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:02.010803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:02.010994 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:02.011528 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:37:02.011535 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:37:02.011893 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 06:37:02.012156 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 06:37:02.012716 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":304,"threads_started":1}
I20260619 06:37:02.015532 11450 heartbeater.cc:347] Connected to a master server at 127.28.104.254:45577
I20260619 06:37:02.015841 11450 heartbeater.cc:464] Registering TS with master...
I20260619 06:37:02.016571 11450 heartbeater.cc:511] Master 127.28.104.254:45577 requested a full tablet report, sending...
I20260619 06:37:02.018151 11131 ts_manager.cc:194] Registered new tserver with Master: f937264584484ea2b7e172d82ef7eb8c (127.28.104.194:39255)
I20260619 06:37:02.019308 11131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:42925
W20260619 06:37:02.324434 11453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:02.324761 11453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:02.324926 11453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:02.332645 11453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:02.332760 11453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:02.333010 11453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:37:02.342929 11453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:45871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=38777
--tserver_master_addrs=127.28.104.254:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:02.344317 11453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:02.345736 11453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:02.359369 11459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:02.359269 11461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:02.359372 11458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:02.360594 11453 server_base.cc:1034] running on GCE node
I20260619 06:37:02.361222 11453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:02.362180 11453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:02.363407 11453 hybrid_clock.cc:648] HybridClock initialized: now 1781851022363361 us; error 55 us; skew 500 ppm
I20260619 06:37:02.363734 11453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:02 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:02.366832 11453 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:37:02.368177 11453 webserver.cc:460] Webserver started at http://127.28.104.195:38777/ using document root <none> and password file <none>
I20260619 06:37:02.368854 11453 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:02.369009 11453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:02.374311 11453 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:37:02.376996 11468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:02.378324 11453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:37:02.378496 11453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "7c506a1327004d3fb3e2b635887f81a0"
format_stamp: "Formatted at 2026-06-19 06:36:59 on dist-test-slave-nf4n"
server_key: "81f154cb55bfd11c2b71fdcaa6227e7b"
server_key_iv: "b9dc5507f1c810bf01d232ce4a588cad"
server_key_version: "encryptionkey@0"
I20260619 06:37:02.379439 11453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:02.401294 11453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:02.405171 11453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:02.405438 11453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:02.406407 11453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:02.407757 11453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:02.407881 11453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:02.408012 11453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:02.408100 11453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:02.440794 11453 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:45871
I20260619 06:37:02.440878 11582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:45871 every 8 connection(s)
I20260619 06:37:02.442276 11453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:37:02.448295 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 11453
Jun 19 06:37:02 dist-test-slave-nf4n krb5kdc[10543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:37:02.464748 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.444932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:57725 (local address 127.28.104.254:45577)
0619 06:37:02.445166 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.445173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.446321 (+ 1148us) server_negotiation.cc:407] Connection header received
0619 06:37:02.448075 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.448081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.448172 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.448282 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:02.450147 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.451344 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.453630 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.454299 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.454430 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.458153 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:02.458161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:02.458167 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:02.458211 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:02.461363 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:02.462234 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:02.462238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:02.462240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:02.462390 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:02.462803 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:02.462809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:02.462813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:02.463147 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:02.463284 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.464151 (+ 867us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.464372 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:37:02.465162 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.449190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.28.104.254:45577)
0619 06:37:02.449642 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.449648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.449999 (+ 351us) server_negotiation.cc:407] Connection header received
0619 06:37:02.450139 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.450145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.450218 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.450305 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:02.452080 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.453277 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.455076 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.455762 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.455890 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.458903 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:02.458911 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:02.458918 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:02.458973 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:02.462296 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:02.463042 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:02.463046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:02.463048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:02.463163 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:02.463582 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:02.463587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:02.463590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:02.463913 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:02.464172 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.464516 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.464749 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":247,"threads_started":1}
I20260619 06:37:02.466040 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.445481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:57725)
0619 06:37:02.446164 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:02.446200 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:02.447805 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:02.448556 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:02.448586 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:02.449206 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:37:02.449929 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.449945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.451546 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.451550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.453504 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.453515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.454505 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.454509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.454696 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.455359 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:37:02.455395 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:37:02.457918 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:37:02.461497 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:02.461505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:02.461523 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:02.462113 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:02.462501 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:02.462505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:02.462507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:02.462680 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:02.463289 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:37:02.463300 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:37:02.463710 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 06:37:02.464019 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 06:37:02.464566 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":257,"threads_started":1}
I20260619 06:37:02.467437 11583 heartbeater.cc:347] Connected to a master server at 127.28.104.254:45577
I20260619 06:37:02.467757 11583 heartbeater.cc:464] Registering TS with master...
I20260619 06:37:02.468467 11583 heartbeater.cc:511] Master 127.28.104.254:45577 requested a full tablet report, sending...
I20260619 06:37:02.469895 11131 ts_manager.cc:194] Registered new tserver with Master: 7c506a1327004d3fb3e2b635887f81a0 (127.28.104.195:45871)
I20260619 06:37:02.470638 11131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:57725
I20260619 06:37:02.480146 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:37:02.487540 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.481184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.28.104.254:45577)
0619 06:37:02.481364 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.481368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.481677 (+ 309us) server_negotiation.cc:407] Connection header received
0619 06:37:02.481865 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.481868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.482015 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.482136 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:37:02.482597 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.483338 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.484967 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.485523 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.485628 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.486216 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:37:02.486799 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:37:02.486917 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.487148 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.487267 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:37:02.488536 11131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:37:02.495921 11131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:37:02.521970 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.507822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:41021 (local address 127.0.0.1:39634)
0619 06:37:02.508706 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:02.509263 (+ 557us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:02.509486 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:02.510461 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:02.510471 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:02.510514 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:02.511056 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.511073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.513919 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.513928 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.517264 (+ 3336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.517289 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.520510 (+ 3221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.520519 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.520851 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.520941 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 06:37:02.521107 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 06:37:02.521409 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":359,"threads_started":1}
I20260619 06:37:02.524166 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.508044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.28.104.193:41021)
0619 06:37:02.509162 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.509168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.509721 (+ 553us) server_negotiation.cc:407] Connection header received
0619 06:37:02.509847 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.509858 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.510180 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.511326 (+ 1146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:02.511637 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.513710 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.517981 (+ 4271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.520182 (+ 2201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.523125 (+ 2943us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.523393 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.523486 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.523750 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":851,"threads_started":1}
I20260619 06:37:02.530396 11252 tablet_service.cc:1467] Processing CreateTablet for tablet 0897ae8fcb7a4726af6e53a813df9a81 (DEFAULT_TABLE table=test-table [id=8bba9fa6d8db46339719a08801194847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:37:02.532163 11252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0897ae8fcb7a4726af6e53a813df9a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:37:02.532567 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.508494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:39255 (local address 127.0.0.1:40240)
0619 06:37:02.522328 (+ 13834us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:02.522355 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:02.522505 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:02.525536 (+ 3031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:02.525542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:02.525576 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:02.525909 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.525922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.528092 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.528097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.530050 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.530065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.531823 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.531830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.532025 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.532062 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:37:02.532126 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 06:37:02.532272 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13686,"thread_start_us":19088,"threads_started":1}
I20260619 06:37:02.533686 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.509136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.28.104.194:39255)
0619 06:37:02.524514 (+ 15378us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.524526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.524572 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:37:02.524705 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.524718 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.525038 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.526115 (+ 1077us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:02.526374 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.527895 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.530235 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.531645 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.532981 (+ 1336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.533172 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.533239 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.533408 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15163,"thread_start_us":16103,"threads_started":1}
I20260619 06:37:02.537899 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.510326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.28.104.195:45871)
0619 06:37:02.525329 (+ 15003us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.525341 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.526277 (+ 936us) server_negotiation.cc:407] Connection header received
0619 06:37:02.526408 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.526421 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.526761 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.526991 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:02.529385 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.531493 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.534693 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.536458 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.536625 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.536802 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.537387 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.537547 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14758,"thread_start_us":14518,"threads_started":1}
I20260619 06:37:02.537449 11385 tablet_service.cc:1467] Processing CreateTablet for tablet 0897ae8fcb7a4726af6e53a813df9a81 (DEFAULT_TABLE table=test-table [id=8bba9fa6d8db46339719a08801194847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:37:02.538569 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.509661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:45871 (local address 127.0.0.1:42362)
0619 06:37:02.525964 (+ 16303us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:02.525988 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:02.526134 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:02.527206 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:02.527210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:02.527235 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:02.529168 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.529183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.532502 (+ 3319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.532507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.534489 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.534507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.536996 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.537002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.537182 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.537214 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:37:02.538093 (+ 879us) client_negotiation.cc:240] Negotiation successful
0619 06:37:02.538245 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16129,"thread_start_us":15856,"threads_started":1}
I20260619 06:37:02.538985 11385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0897ae8fcb7a4726af6e53a813df9a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:37:02.542490 11516 tablet_service.cc:1467] Processing CreateTablet for tablet 0897ae8fcb7a4726af6e53a813df9a81 (DEFAULT_TABLE table=test-table [id=8bba9fa6d8db46339719a08801194847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:37:02.543916 11516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0897ae8fcb7a4726af6e53a813df9a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:37:02.546923 11599 tablet_bootstrap.cc:492] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: Bootstrap starting.
I20260619 06:37:02.550699 11599 tablet_bootstrap.cc:654] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: Neither blocks nor log segments found. Creating new log.
I20260619 06:37:02.551867 11599 log.cc:824] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:02.555454 11599 tablet_bootstrap.cc:492] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: No bootstrap required, opened a new log
I20260619 06:37:02.555630 11601 tablet_bootstrap.cc:492] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: Bootstrap starting.
I20260619 06:37:02.555917 11599 ts_tablet_manager.cc:1397] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:37:02.559131 11602 tablet_bootstrap.cc:492] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: Bootstrap starting.
I20260619 06:37:02.559362 11601 tablet_bootstrap.cc:654] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: Neither blocks nor log segments found. Creating new log.
I20260619 06:37:02.560608 11601 log.cc:824] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:02.562441 11602 tablet_bootstrap.cc:654] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: Neither blocks nor log segments found. Creating new log.
I20260619 06:37:02.563580 11602 log.cc:824] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:02.564839 11601 tablet_bootstrap.cc:492] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: No bootstrap required, opened a new log
I20260619 06:37:02.565152 11601 ts_tablet_manager.cc:1397] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260619 06:37:02.565809 11317 heartbeater.cc:503] Master 127.28.104.254:45577 was elected leader, sending a full tablet report...
I20260619 06:37:02.566465 11602 tablet_bootstrap.cc:492] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: No bootstrap required, opened a new log
I20260619 06:37:02.566720 11602 ts_tablet_manager.cc:1397] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 06:37:02.567190 11599 raft_consensus.cc:348] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } }
I20260619 06:37:02.567607 11599 raft_consensus.cc:374] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:37:02.567699 11599 raft_consensus.cc:729] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01bc3be45574d368cbe6efc120fd6b1, State: Initialized, Role: FOLLOWER
I20260619 06:37:02.568365 11599 consensus_queue.cc:260] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } }
I20260619 06:37:02.574492 11599 ts_tablet_manager.cc:1428] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260619 06:37:02.576081 11602 raft_consensus.cc:348] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } }
I20260619 06:37:02.576601 11602 raft_consensus.cc:374] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:37:02.576731 11602 raft_consensus.cc:729] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c506a1327004d3fb3e2b635887f81a0, State: Initialized, Role: FOLLOWER
I20260619 06:37:02.577409 11602 consensus_queue.cc:260] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } }
I20260619 06:37:02.578131 11601 raft_consensus.cc:348] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } }
I20260619 06:37:02.578514 11601 raft_consensus.cc:374] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:37:02.578562 11583 heartbeater.cc:503] Master 127.28.104.254:45577 was elected leader, sending a full tablet report...
I20260619 06:37:02.578625 11601 raft_consensus.cc:729] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f937264584484ea2b7e172d82ef7eb8c, State: Initialized, Role: FOLLOWER
I20260619 06:37:02.579274 11602 ts_tablet_manager.cc:1428] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260619 06:37:02.579216 11601 consensus_queue.cc:260] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } }
I20260619 06:37:02.580291 11450 heartbeater.cc:503] Master 127.28.104.254:45577 was elected leader, sending a full tablet report...
I20260619 06:37:02.581045 11601 ts_tablet_manager.cc:1428] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
I20260619 06:37:02.664317 11607 raft_consensus.cc:482] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:37:02.664639 11607 raft_consensus.cc:504] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } }
I20260619 06:37:02.666396 11607 leader_election.cc:288] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01bc3be45574d368cbe6efc120fd6b1 (127.28.104.193:41021), 7c506a1327004d3fb3e2b635887f81a0 (127.28.104.195:45871)
I20260619 06:37:02.669983 11606 raft_consensus.cc:482] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:37:02.670302 11606 raft_consensus.cc:504] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } }
I20260619 06:37:02.674175 11606 leader_election.cc:288] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01bc3be45574d368cbe6efc120fd6b1 (127.28.104.193:41021), f937264584484ea2b7e172d82ef7eb8c (127.28.104.194:39255)
I20260619 06:37:02.675257 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.667766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:42577 (local address 127.28.104.195:45871)
0619 06:37:02.667968 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.667973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.668304 (+ 331us) server_negotiation.cc:407] Connection header received
0619 06:37:02.668504 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.668508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.668589 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.668679 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:02.669236 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.670283 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.672684 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.674097 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.674257 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.674332 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.674858 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.674987 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:37:02.675981 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.667552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:45871 (local address 127.28.104.194:42577)
0619 06:37:02.668180 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:02.668203 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:02.668348 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:02.668705 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:02.668708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:02.668728 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:02.668994 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.669001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.670430 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.670436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.672497 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.672510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.674488 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.674494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.674661 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.674694 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:37:02.675419 (+ 725us) client_negotiation.cc:240] Negotiation successful
0619 06:37:02.675566 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":32,"thread_start_us":326,"threads_started":1}
I20260619 06:37:02.676491 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.666838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:46677 (local address 127.28.104.193:41021)
0619 06:37:02.667029 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.667033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.667547 (+ 514us) server_negotiation.cc:407] Connection header received
0619 06:37:02.667762 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.667767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.667851 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.667960 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:02.668649 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.669639 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.672234 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.673604 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.673740 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.673815 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.676044 (+ 2229us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.676202 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:37:02.677292 11537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0897ae8fcb7a4726af6e53a813df9a81" candidate_uuid: "f937264584484ea2b7e172d82ef7eb8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c506a1327004d3fb3e2b635887f81a0" is_pre_election: true
I20260619 06:37:02.677723 11537 raft_consensus.cc:2454] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f937264584484ea2b7e172d82ef7eb8c in term 0.
I20260619 06:37:02.678076 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.666734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:41021 (local address 127.28.104.194:46677)
0619 06:37:02.667435 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:02.667456 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:02.667593 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:02.668065 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:02.668070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:02.668100 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:02.668457 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.668465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.669778 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.669783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.672029 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.672044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.674016 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.674022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.675840 (+ 1818us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.675879 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:37:02.677424 (+ 1545us) client_negotiation.cc:240] Negotiation successful
0619 06:37:02.677670 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":141,"thread_start_us":371,"threads_started":1}
I20260619 06:37:02.678699 11338 leader_election.cc:302] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c506a1327004d3fb3e2b635887f81a0, f937264584484ea2b7e172d82ef7eb8c; no voters:
I20260619 06:37:02.679131 11607 raft_consensus.cc:2781] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:37:02.679040 11272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0897ae8fcb7a4726af6e53a813df9a81" candidate_uuid: "f937264584484ea2b7e172d82ef7eb8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01bc3be45574d368cbe6efc120fd6b1" is_pre_election: true
I20260619 06:37:02.679291 11607 raft_consensus.cc:482] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:37:02.679385 11607 raft_consensus.cc:3037] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:37:02.679634 11272 raft_consensus.cc:2454] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f937264584484ea2b7e172d82ef7eb8c in term 0.
I20260619 06:37:02.680145 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.672109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:39255 (local address 127.28.104.195:55545)
0619 06:37:02.672446 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:02.672465 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:02.672570 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:02.673473 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:02.673480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:02.673513 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:02.673830 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.673843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.675285 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.675291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.677104 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.677116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.679404 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.679410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.679605 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.679639 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:37:02.679736 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:37:02.679858 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":3}
I20260619 06:37:02.680850 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.672771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:55545 (local address 127.28.104.194:39255)
0619 06:37:02.672988 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.672996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.673037 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:37:02.673101 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.673109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.673197 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.673289 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:02.674014 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.675117 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.677243 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.679128 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.679258 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.679325 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.680427 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.680565 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":78}
I20260619 06:37:02.681660 11405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0897ae8fcb7a4726af6e53a813df9a81" candidate_uuid: "7c506a1327004d3fb3e2b635887f81a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f937264584484ea2b7e172d82ef7eb8c" is_pre_election: true
I20260619 06:37:02.682557 11610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.672337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:58103 (local address 127.28.104.193:41021)
0619 06:37:02.675539 (+ 3202us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.675547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.675571 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:37:02.675631 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.675636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.675706 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.676418 (+ 712us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:02.676738 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.677900 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.680246 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.681563 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.681747 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.681829 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.682090 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.682222 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3090,"thread_start_us":2895,"threads_started":1}
I20260619 06:37:02.682569 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.672128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:41021 (local address 127.28.104.195:58103)
0619 06:37:02.674821 (+ 2693us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:02.674843 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:02.674976 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:02.675854 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:02.675859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:02.675886 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:02.676257 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.676274 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.678079 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.678084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.680051 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:02.680063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.681761 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.681768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:02.681930 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.681963 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:37:02.682048 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:37:02.682175 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2535,"mutex_wait_us":480,"thread_start_us":1866,"threads_started":1}
I20260619 06:37:02.682484 11607 raft_consensus.cc:504] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } }
I20260619 06:37:02.682962 11405 raft_consensus.cc:2379] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c506a1327004d3fb3e2b635887f81a0 in current term 1: Already voted for candidate f937264584484ea2b7e172d82ef7eb8c in this term.
I20260619 06:37:02.683224 11607 leader_election.cc:288] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [CANDIDATE]: Term 1 election: Requested vote from peers b01bc3be45574d368cbe6efc120fd6b1 (127.28.104.193:41021), 7c506a1327004d3fb3e2b635887f81a0 (127.28.104.195:45871)
I20260619 06:37:02.683262 11272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0897ae8fcb7a4726af6e53a813df9a81" candidate_uuid: "7c506a1327004d3fb3e2b635887f81a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01bc3be45574d368cbe6efc120fd6b1" is_pre_election: true
I20260619 06:37:02.683513 11272 raft_consensus.cc:2454] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c506a1327004d3fb3e2b635887f81a0 in term 0.
I20260619 06:37:02.683961 11272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0897ae8fcb7a4726af6e53a813df9a81" candidate_uuid: "f937264584484ea2b7e172d82ef7eb8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01bc3be45574d368cbe6efc120fd6b1"
I20260619 06:37:02.684063 11537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0897ae8fcb7a4726af6e53a813df9a81" candidate_uuid: "f937264584484ea2b7e172d82ef7eb8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c506a1327004d3fb3e2b635887f81a0"
I20260619 06:37:02.684167 11272 raft_consensus.cc:3037] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:37:02.684123 11469 leader_election.cc:302] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c506a1327004d3fb3e2b635887f81a0, b01bc3be45574d368cbe6efc120fd6b1; no voters: f937264584484ea2b7e172d82ef7eb8c
I20260619 06:37:02.684288 11537 raft_consensus.cc:3037] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:37:02.686546 11272 raft_consensus.cc:2454] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f937264584484ea2b7e172d82ef7eb8c in term 1.
I20260619 06:37:02.686959 11537 raft_consensus.cc:2454] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f937264584484ea2b7e172d82ef7eb8c in term 1.
I20260619 06:37:02.687005 11338 leader_election.cc:302] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01bc3be45574d368cbe6efc120fd6b1, f937264584484ea2b7e172d82ef7eb8c; no voters:
I20260619 06:37:02.687275 11606 raft_consensus.cc:2741] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 06:37:02.687318 11607 raft_consensus.cc:2781] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:37:02.687772 11607 raft_consensus.cc:686] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 1 LEADER]: Becoming Leader. State: Replica: f937264584484ea2b7e172d82ef7eb8c, State: Running, Role: LEADER
I20260619 06:37:02.688257 11607 consensus_queue.cc:237] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } }
I20260619 06:37:02.692584 11131 catalog_manager.cc:5495] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c reported cstate change: term changed from 0 to 1, leader changed from <none> to f937264584484ea2b7e172d82ef7eb8c (127.28.104.194). New cstate: current_term: 1 leader_uuid: "f937264584484ea2b7e172d82ef7eb8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f937264584484ea2b7e172d82ef7eb8c" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 39255 } health_report { overall_health: HEALTHY } } }
W20260619 06:37:02.693779 11584 tablet.cc:2369] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:37:02.733206 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.728104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.28.104.194:39255)
0619 06:37:02.728261 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.728266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.728283 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:37:02.728458 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.728462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.728548 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.728668 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:37:02.728998 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.729893 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.731270 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.731653 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.731745 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.732162 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:37:02.732658 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:37:02.732764 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.732844 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.732903 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:37:02.741321 11537 raft_consensus.cc:1261] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 1 FOLLOWER]: Refusing update from remote peer f937264584484ea2b7e172d82ef7eb8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:37:02.741405 11272 raft_consensus.cc:1261] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [term 1 FOLLOWER]: Refusing update from remote peer f937264584484ea2b7e172d82ef7eb8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:37:02.742206 11612 consensus_queue.cc:1035] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c506a1327004d3fb3e2b635887f81a0" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:37:02.742558 11607 consensus_queue.cc:1035] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01bc3be45574d368cbe6efc120fd6b1" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:37:02.743919 11451 tablet.cc:2369] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:37:02.754020 11614 mvcc.cc:205] Tried to move back new op lower bound from 7298461789138284544 to 7298461788935503872. Current Snapshot: MvccSnapshot[applied={T|T < 7298461789138284544}]
I20260619 06:37:02.769181 11129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:37:02.769505 11129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:37:02.774116 11129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Sending DeleteTablet for 3 replicas of tablet 0897ae8fcb7a4726af6e53a813df9a81
I20260619 06:37:02.775724 11516 tablet_service.cc:1514] Processing DeleteTablet for tablet 0897ae8fcb7a4726af6e53a813df9a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:37:02 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:42362
I20260619 06:37:02.775749 11252 tablet_service.cc:1514] Processing DeleteTablet for tablet 0897ae8fcb7a4726af6e53a813df9a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:37:02 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:39634
I20260619 06:37:02.778694 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 11168
I20260619 06:37:02.780064 11625 tablet_replica.cc:330] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: stopping tablet replica
I20260619 06:37:02.780512 11625 raft_consensus.cc:2229] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:37:02.781081 11625 raft_consensus.cc:2258] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:37:02.783416 11625 ts_tablet_manager.cc:1905] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:02.783711 11385 tablet_service.cc:1514] Processing DeleteTablet for tablet 0897ae8fcb7a4726af6e53a813df9a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:37:02 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:40240
I20260619 06:37:02.789561 11624 tablet_replica.cc:330] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: stopping tablet replica
I20260619 06:37:02.790172 11624 raft_consensus.cc:2229] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:37:02.790735 11624 raft_consensus.cc:2258] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:37:02.790838 11625 ts_tablet_manager.cc:1918] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:02.791011 11625 log.cc:1196] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/0897ae8fcb7a4726af6e53a813df9a81
I20260619 06:37:02.791599 11625 ts_tablet_manager.cc:1939] T 0897ae8fcb7a4726af6e53a813df9a81 P b01bc3be45574d368cbe6efc120fd6b1: Deleting consensus metadata
I20260619 06:37:02.793641 11115 catalog_manager.cc:4841] TS b01bc3be45574d368cbe6efc120fd6b1 (127.28.104.193:41021): tablet 0897ae8fcb7a4726af6e53a813df9a81 (table test-table [id=8bba9fa6d8db46339719a08801194847]) successfully deleted
I20260619 06:37:02.798089 11626 tablet_replica.cc:330] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: stopping tablet replica
I20260619 06:37:02.798213 11624 ts_tablet_manager.cc:1905] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:02.798811 11626 raft_consensus.cc:2229] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:37:02.799722 11626 raft_consensus.cc:2258] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:37:02.802251 11626 ts_tablet_manager.cc:1905] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:02.803287 11610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.779519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39644 (local address 127.28.104.193:41021)
0619 06:37:02.783006 (+ 3487us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.783018 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.783064 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:37:02.783158 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.783168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.783287 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.784845 (+ 1558us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:02.786675 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.788385 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.790306 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.790983 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.791233 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.795735 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:02.795749 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:02.795768 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:02.795815 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:02.799805 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:02.800765 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:02.800775 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:02.800779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:02.800955 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:02.801482 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:02.801488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:02.801492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:02.802004 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:02.802288 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.802688 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.802950 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3288}
I20260619 06:37:02.804234 11312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:02.808702 11624 ts_tablet_manager.cc:1918] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:02.808904 11624 log.cc:1196] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/0897ae8fcb7a4726af6e53a813df9a81
I20260619 06:37:02.809499 11624 ts_tablet_manager.cc:1939] T 0897ae8fcb7a4726af6e53a813df9a81 P 7c506a1327004d3fb3e2b635887f81a0: Deleting consensus metadata
I20260619 06:37:02.810130 11626 ts_tablet_manager.cc:1918] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:02.810343 11626 log.cc:1196] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/0897ae8fcb7a4726af6e53a813df9a81
I20260619 06:37:02.811015 11626 ts_tablet_manager.cc:1939] T 0897ae8fcb7a4726af6e53a813df9a81 P f937264584484ea2b7e172d82ef7eb8c: Deleting consensus metadata
I20260619 06:37:02.811673 11115 catalog_manager.cc:4841] TS 7c506a1327004d3fb3e2b635887f81a0 (127.28.104.195:45871): tablet 0897ae8fcb7a4726af6e53a813df9a81 (table test-table [id=8bba9fa6d8db46339719a08801194847]) successfully deleted
I20260619 06:37:02.813002 11118 catalog_manager.cc:4841] TS f937264584484ea2b7e172d82ef7eb8c (127.28.104.194:39255): tablet 0897ae8fcb7a4726af6e53a813df9a81 (table test-table [id=8bba9fa6d8db46339719a08801194847]) successfully deleted
I20260619 06:37:02.903223 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 11168
I20260619 06:37:02.928052 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 11321
I20260619 06:37:02.942406 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:02.928853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.28.104.194:39255)
0619 06:37:02.929159 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:02.929165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:02.929185 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:37:02.929384 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:02.929390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:02.929466 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:02.929580 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:02.931029 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.931798 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.933148 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:02.933588 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:02.933727 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:02.936304 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:02.936328 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:02.936347 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:02.936381 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:02.939494 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:02.940184 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:02.940196 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:02.940203 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:02.940350 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:02.940765 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:02.940772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:02.940781 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:02.941237 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:02.941551 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:02.941836 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 06:37:02.942062 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:37:02.943369 11445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:03.057852 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 11321
I20260619 06:37:03.077325 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 11453
I20260619 06:37:03.091785 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:03.078088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.28.104.195:45871)
0619 06:37:03.078316 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:03.078323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:03.078371 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:37:03.078592 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:03.078598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:03.078683 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:03.078793 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:03.080304 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:03.081091 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:03.082459 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:03.082867 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:03.083025 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:03.085625 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:03.085647 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:03.085663 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:03.085694 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:03.088597 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:03.089688 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:03.089696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:03.089699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:03.089839 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:03.090225 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:03.090230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:03.090234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:03.090593 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:03.090891 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:03.091213 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 06:37:03.091426 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260619 06:37:03.092521 11578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:03.183609 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 11453
I20260619 06:37:03.200764 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 11098
I20260619 06:37:03.201929 11161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:03.314033 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 11098
I20260619 06:37:03.335819 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:45577
--webserver_interface=127.28.104.254
--webserver_port=39003
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:45577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:37:03.661062 11631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:03.661350 11631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:03.661530 11631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:03.668820 11631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:37:03.668921 11631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:03.669047 11631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:03.669137 11631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:37:03.669216 11631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:37:03.679045 11631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:45577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:45577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=39003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:03.680346 11631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:03.681694 11631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:03.693686 11637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:03.693737 11640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:03.693764 11638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:03.694823 11631 server_base.cc:1034] running on GCE node
I20260619 06:37:03.695541 11631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:03.696843 11631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:03.698069 11631 hybrid_clock.cc:648] HybridClock initialized: now 1781851023698007 us; error 60 us; skew 500 ppm
I20260619 06:37:03.698429 11631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:03 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:03.701615 11631 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:37:03.703020 11631 webserver.cc:460] Webserver started at http://127.28.104.254:39003/ using document root <none> and password file <none>
I20260619 06:37:03.703699 11631 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:03.703845 11631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:03.709131 11631 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:37:03.711930 11647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:03.713337 11631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:37:03.713505 11631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "845fdc93e413458ba83be72fbc72fb00"
format_stamp: "Formatted at 2026-06-19 06:36:57 on dist-test-slave-nf4n"
server_key: "d619a2a36b8edcafaa482a8256d6a00d"
server_key_iv: "c3d5d16fd79ca64ad37354422d42be75"
server_key_version: "encryptionkey@0"
I20260619 06:37:03.714560 11631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:03.743103 11631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:03.747547 11631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:03.747825 11631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:03.766314 11631 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:45577
I20260619 06:37:03.766340 11698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:45577 every 8 connection(s)
I20260619 06:37:03.767818 11631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:37:03.774745 11699 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:37:03.777961 11699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Bootstrap starting.
I20260619 06:37:03.777974 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 11631
I20260619 06:37:03.778798 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:41021
--local_ip_for_outbound_sockets=127.28.104.193
--tserver_master_addrs=127.28.104.254:45577
--webserver_port=40623
--webserver_interface=127.28.104.193
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:37:03.798405 11699 log.cc:824] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:03.814728 11699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:37:03.815276 11699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00: Bootstrap complete.
I20260619 06:37:03.824838 11699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:37:03.825794 11699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845fdc93e413458ba83be72fbc72fb00, State: Initialized, Role: FOLLOWER
I20260619 06:37:03.826503 11699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:37:03.826766 11699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:37:03.826880 11699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:37:03.827077 11699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:37:03.831148 11699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:37:03.831694 11699 leader_election.cc:302] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 845fdc93e413458ba83be72fbc72fb00; no voters:
I20260619 06:37:03.832382 11699 leader_election.cc:288] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 06:37:03.832515 11703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:37:03.833735 11703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [term 3 LEADER]: Becoming Leader. State: Replica: 845fdc93e413458ba83be72fbc72fb00, State: Running, Role: LEADER
I20260619 06:37:03.834384 11703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } }
I20260619 06:37:03.834690 11699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:37:03.836992 11705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 845fdc93e413458ba83be72fbc72fb00. Latest consensus state: current_term: 3 leader_uuid: "845fdc93e413458ba83be72fbc72fb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } } }
I20260619 06:37:03.837411 11705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: This master's current role is: LEADER
I20260619 06:37:03.837224 11704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "845fdc93e413458ba83be72fbc72fb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "845fdc93e413458ba83be72fbc72fb00" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 45577 } } }
I20260619 06:37:03.837610 11704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 845fdc93e413458ba83be72fbc72fb00 [sys.catalog]: This master's current role is: LEADER
I20260619 06:37:03.847203 11710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:37:03.857991 11710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:37:03.860002 11710 catalog_manager.cc:1259] Loaded cluster ID: 9cb0b7929c074f9680e9710ed155eaaa
I20260619 06:37:03.860216 11710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 06:37:03.863093 11710 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f72fd4a39ad google::LogMessage::Fail() at ??:0
@ 0x7f72fd4a688a google::LogMessage::SendToLog() at ??:0
@ 0x7f72fd4a34de google::LogMessage::Flush() at ??:0
@ 0x7f72fd4a4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7312ed6512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7312ed3976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7312f5b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7312f5b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f73000ef345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f73000f291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f73000f2799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f73000cdeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73033a76db start_thread at ??:0
@ 0x7f72fc33d61f clone at ??:0
W20260619 06:37:04.151605 11701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:04.151944 11701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:04.152113 11701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:04.159513 11701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:04.159673 11701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:04.159907 11701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:37:04.170215 11701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:41021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=40623
--tserver_master_addrs=127.28.104.254:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:04.171511 11701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:04.172845 11701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:04.185680 11741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:04.185762 11744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:04.185727 11742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:04.186807 11701 server_base.cc:1034] running on GCE node
I20260619 06:37:04.187410 11701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:04.188274 11701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:04.189465 11701 hybrid_clock.cc:648] HybridClock initialized: now 1781851024189427 us; error 29 us; skew 500 ppm
I20260619 06:37:04.189747 11701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:04 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:04.193109 11701 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:37:04.194684 11701 webserver.cc:460] Webserver started at http://127.28.104.193:40623/ using document root <none> and password file <none>
I20260619 06:37:04.195417 11701 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:04.195580 11701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:04.201298 11701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:37:04.204270 11751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:04.205801 11701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:37:04.205986 11701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "b01bc3be45574d368cbe6efc120fd6b1"
format_stamp: "Formatted at 2026-06-19 06:36:58 on dist-test-slave-nf4n"
server_key: "2fec02a489296f3ed4ad61c8ae25d043"
server_key_iv: "1d4c54b9e5487ac4ed8c0083197c85fc"
server_key_version: "encryptionkey@0"
I20260619 06:37:04.206984 11701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:04.228140 11701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:04.232528 11701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:04.232826 11701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:04.233853 11701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:04.235164 11701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:04.235292 11701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:04.235447 11701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:04.235551 11701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:04.269203 11701 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:41021
I20260619 06:37:04.269289 11863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:41021 every 8 connection(s)
I20260619 06:37:04.270630 11701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:37:04.272185 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 11701
I20260619 06:37:04.272877 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:39255
--local_ip_for_outbound_sockets=127.28.104.194
--tserver_master_addrs=127.28.104.254:45577
--webserver_port=45339
--webserver_interface=127.28.104.194
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:37:04.277174 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:04.274575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:38325)
0619 06:37:04.275500 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:04.275821 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":326,"threads_started":1}
W20260619 06:37:04.277945 11864 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:45577 (0 consecutive failures): Network error: Failed to ping master at 127.28.104.254:45577: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
I20260619 06:37:04.278949 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:04.278472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:41823)
0619 06:37:04.278677 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:04.278784 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 06:37:04.279981 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:04.279561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:55133)
0619 06:37:04.279757 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:04.279868 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 06:37:04.280268 11864 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:37:04.605553 11867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:04.605840 11867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:04.605994 11867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:04.613404 11867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:04.613512 11867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:04.613718 11867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:37:04.623775 11867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:39255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=45339
--tserver_master_addrs=127.28.104.254:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:04.625166 11867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:04.626461 11867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:04.639460 11876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:04.639532 11872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:04.639487 11873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:04.640687 11867 server_base.cc:1034] running on GCE node
I20260619 06:37:04.641357 11867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:04.642307 11867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:04.643534 11867 hybrid_clock.cc:648] HybridClock initialized: now 1781851024643482 us; error 53 us; skew 500 ppm
I20260619 06:37:04.643849 11867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:04 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:04.649864 11867 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:37:04.651489 11867 webserver.cc:460] Webserver started at http://127.28.104.194:45339/ using document root <none> and password file <none>
I20260619 06:37:04.652240 11867 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:04.652386 11867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:04.657707 11867 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:37:04.660326 11883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:04.661687 11867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:37:04.661867 11867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "f937264584484ea2b7e172d82ef7eb8c"
format_stamp: "Formatted at 2026-06-19 06:36:59 on dist-test-slave-nf4n"
server_key: "f9385c58d7603129722b2d8927867325"
server_key_iv: "de7000cb9aba27595635e9c0c09a7dc0"
server_key_version: "encryptionkey@0"
I20260619 06:37:04.662849 11867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:04.677537 11867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:04.681591 11867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:04.681886 11867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:04.682874 11867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:04.684213 11867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:04.684332 11867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:04.684474 11867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:04.684561 11867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:04.718269 11867 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:39255
I20260619 06:37:04.718353 11995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:39255 every 8 connection(s)
I20260619 06:37:04.719691 11867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:37:04.724455 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:04.722686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:52031)
0619 06:37:04.723424 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:04.723642 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":313,"threads_started":1}
W20260619 06:37:04.725173 11996 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:45577 (0 consecutive failures): Network error: Failed to ping master at 127.28.104.254:45577: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
I20260619 06:37:04.725947 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:04.725649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:34149)
0619 06:37:04.725778 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:04.725849 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 06:37:04.726744 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:04.726444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:44115)
0619 06:37:04.726570 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:04.726642 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 06:37:04.726976 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 11867
W20260619 06:37:04.727037 11996 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:37:04.727605 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:45871
--local_ip_for_outbound_sockets=127.28.104.195
--tserver_master_addrs=127.28.104.254:45577
--webserver_port=38777
--webserver_interface=127.28.104.195
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:37:05.056366 11999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:05.056645 11999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:05.056738 11999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:05.063957 11999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:05.064070 11999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:05.064183 11999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:37:05.074100 11999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:45871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=38777
--tserver_master_addrs=127.28.104.254:45577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:05.075327 11999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:05.076612 11999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:05.089749 12007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:05.089777 12004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:05.090018 12005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:05.090660 11999 server_base.cc:1034] running on GCE node
I20260619 06:37:05.091229 11999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:05.092168 11999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:05.093355 11999 hybrid_clock.cc:648] HybridClock initialized: now 1781851025093301 us; error 55 us; skew 500 ppm
I20260619 06:37:05.093616 11999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:05 dist-test-slave-nf4n krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:05.096683 11999 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:37:05.098239 11999 webserver.cc:460] Webserver started at http://127.28.104.195:38777/ using document root <none> and password file <none>
I20260619 06:37:05.098920 11999 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:05.099052 11999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:05.104235 11999 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:37:05.106856 12014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:05.108129 11999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:37:05.108309 11999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "7c506a1327004d3fb3e2b635887f81a0"
format_stamp: "Formatted at 2026-06-19 06:36:59 on dist-test-slave-nf4n"
server_key: "81f154cb55bfd11c2b71fdcaa6227e7b"
server_key_iv: "b9dc5507f1c810bf01d232ce4a588cad"
server_key_version: "encryptionkey@0"
I20260619 06:37:05.109359 11999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:05.128684 11999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:05.132727 11999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:05.133062 11999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:05.134061 11999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:05.135351 11999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:05.135475 11999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:05.135622 11999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:05.135715 11999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:05.168589 11999 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:45871
I20260619 06:37:05.168651 12127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:45871 every 8 connection(s)
I20260619 06:37:05.170346 11999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:37:05.175130 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:05.173427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:43255)
0619 06:37:05.174108 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:05.174312 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":271,"threads_started":1}
W20260619 06:37:05.175745 12128 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:45577 (0 consecutive failures): Network error: Failed to ping master at 127.28.104.254:45577: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
I20260619 06:37:05.176472 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:05.176186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:56879)
0619 06:37:05.176310 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:05.176377 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:37:05.177234 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:05.176932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:51381)
0619 06:37:05.177075 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:05.177140 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 06:37:05.177467 12128 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:37:05.179279 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 11999
I20260619 06:37:05.281710 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:05.280861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:60767)
0619 06:37:05.281370 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:05.281481 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":302,"threads_started":1}
I20260619 06:37:05.728852 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:05.727558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:53691)
0619 06:37:05.728327 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:05.728508 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":356,"threads_started":1}
I20260619 06:37:06.178745 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:06.177966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:42133)
0619 06:37:06.178404 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:06.178527 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":217,"threads_started":1}
I20260619 06:37:06.283417 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:06.282569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:56521)
0619 06:37:06.283097 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:06.283206 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":224,"threads_started":1}
I20260619 06:37:06.730460 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:06.729750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:46729)
0619 06:37:06.730197 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:06.730283 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":217,"threads_started":1}
I20260619 06:37:07.180341 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:07.179640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:60493)
0619 06:37:07.180080 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:07.180181 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260619 06:37:07.285013 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:07.284317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:44187)
0619 06:37:07.284760 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:07.284838 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
I20260619 06:37:07.732080 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:07.731348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:43961)
0619 06:37:07.731837 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:07.731915 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":287,"threads_started":1}
I20260619 06:37:08.181836 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:08.181165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:42841)
0619 06:37:08.181593 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:08.181671 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":207,"threads_started":1}
I20260619 06:37:08.286674 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:08.285913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:34599)
0619 06:37:08.286403 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:08.286487 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":259,"threads_started":1}
I20260619 06:37:08.734213 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:08.733296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:36455)
0619 06:37:08.733856 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:08.733949 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":310,"threads_started":1}
I20260619 06:37:09.183462 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:09.182774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:60825)
0619 06:37:09.183197 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:09.183276 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":200,"threads_started":1}
I20260619 06:37:09.288369 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:09.287634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:55145)
0619 06:37:09.288094 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:09.288196 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":258,"threads_started":1}
I20260619 06:37:09.736013 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:09.735169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:42693)
0619 06:37:09.735657 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:09.735791 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":238,"threads_started":1}
I20260619 06:37:10.185141 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:10.184317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:35359)
0619 06:37:10.184764 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:10.184844 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":221,"threads_started":1}
I20260619 06:37:10.290225 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:10.289364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:45657)
0619 06:37:10.289935 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:10.290033 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":278,"threads_started":1}
I20260619 06:37:10.737713 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:10.736948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:41985)
0619 06:37:10.737434 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:10.737548 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":276,"threads_started":1}
I20260619 06:37:11.186717 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:11.185976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:50973)
0619 06:37:11.186448 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:11.186555 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
I20260619 06:37:11.292106 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:11.291236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:48537)
0619 06:37:11.291766 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:11.291869 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":219,"threads_started":1}
I20260619 06:37:11.739465 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:11.738627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:39749)
0619 06:37:11.739161 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:11.739248 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":221,"threads_started":1}
I20260619 06:37:12.188827 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:12.187594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:60539)
0619 06:37:12.188319 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:12.188427 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":353,"threads_started":1}
I20260619 06:37:12.293898 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:12.293103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:50617)
0619 06:37:12.293599 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:12.293700 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":229,"threads_started":1}
I20260619 06:37:12.741052 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:12.740343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:49375)
0619 06:37:12.740781 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:12.740863 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":226,"threads_started":1}
I20260619 06:37:13.190483 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:13.189734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:37117)
0619 06:37:13.190230 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:13.190314 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":245,"threads_started":1}
I20260619 06:37:13.295495 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:13.294731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:50935)
0619 06:37:13.295221 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:13.295308 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":240,"threads_started":1}
I20260619 06:37:13.742596 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:13.741906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:38249)
0619 06:37:13.742344 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:13.742422 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":213,"threads_started":1}
I20260619 06:37:14.192210 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:14.191419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:55471)
0619 06:37:14.191930 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:14.192017 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":264,"threads_started":1}
I20260619 06:37:14.297124 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:14.296380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:42147)
0619 06:37:14.296844 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:14.296926 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260619 06:37:14.744277 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:14.743542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:34269)
0619 06:37:14.744019 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:14.744105 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":263,"threads_started":1}
I20260619 06:37:15.193990 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:15.193269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:36313)
0619 06:37:15.193738 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:15.193825 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":265,"threads_started":1}
I20260619 06:37:15.298767 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:15.298088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:45919)
0619 06:37:15.298510 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:15.298596 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":208,"threads_started":1}
I20260619 06:37:15.746004 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:15.745260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:34745)
0619 06:37:15.745727 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:15.745824 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":232,"threads_started":1}
I20260619 06:37:16.195592 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:16.194844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:40705)
0619 06:37:16.195293 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:16.195385 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":228,"threads_started":1}
I20260619 06:37:16.300563 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:16.299834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:48103)
0619 06:37:16.300316 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:16.300402 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":260,"threads_started":1}
I20260619 06:37:16.747668 12166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:16.746912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:48871)
0619 06:37:16.747431 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:16.747511 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":303,"threads_started":1}
I20260619 06:37:17.197640 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:17.196809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:56977)
0619 06:37:17.197407 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:17.197490 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":387,"threads_started":1}
I20260619 06:37:17.302444 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:17.301581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:58231)
0619 06:37:17.302027 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:17.302203 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260619 06:37:17.749286 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:17.748540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:50847)
0619 06:37:17.748989 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:17.749098 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":222,"threads_started":1}
I20260619 06:37:18.199287 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:18.198557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:44013)
0619 06:37:18.199002 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:18.199101 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":215,"threads_started":1}
I20260619 06:37:18.304061 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:18.303365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:35073)
0619 06:37:18.303791 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:18.303876 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":231,"threads_started":1}
I20260619 06:37:18.750931 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:18.750178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:54503)
0619 06:37:18.750589 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:18.750678 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":211,"threads_started":1}
I20260619 06:37:19.200909 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:19.200224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:43783)
0619 06:37:19.200650 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:19.200738 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260619 06:37:19.306149 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:19.305306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.193:46511)
0619 06:37:19.305766 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:19.305878 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
I20260619 06:37:19.752848 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:19.752084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.194:56223)
0619 06:37:19.752556 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:19.752654 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":217,"threads_started":1}
I20260619 06:37:20.183147 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 11701
I20260619 06:37:20.200891 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:20.184423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.28.104.193:41021)
0619 06:37:20.185009 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:20.185069 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:20.185123 (+ 54us) server_negotiation.cc:407] Connection header received
0619 06:37:20.186651 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:20.186660 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:20.187092 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:20.187382 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:20.188699 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:20.190003 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:20.191381 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:20.192060 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:20.192202 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:20.194888 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:20.194897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:20.194912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:20.194948 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:20.197795 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:20.198466 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:20.198472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:20.198479 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:20.198638 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:20.199119 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:20.199122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:20.199124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:20.199598 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:20.199809 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:20.200275 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 06:37:20.200613 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":304,"threads_started":1}
I20260619 06:37:20.202298 11859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:20.203008 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:20.201994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:45577 (local address 127.28.104.195:33235)
0619 06:37:20.202552 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:20.202724 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":268,"threads_started":1}
I20260619 06:37:20.287626 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 11701
I20260619 06:37:20.305100 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 11867
I20260619 06:37:20.324649 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:20.306119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.28.104.194:39255)
0619 06:37:20.306679 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:20.306692 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:20.306728 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:37:20.308286 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:20.308297 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:20.308745 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:20.309163 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:20.310838 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:20.312132 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:20.313778 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:20.315010 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:20.315305 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:20.318205 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:20.318214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:20.318231 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:20.318270 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:20.321564 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:20.322221 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:20.322228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:20.322235 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:20.322406 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:20.322760 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:20.322764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:20.322766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:20.323305 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:20.323561 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:20.323940 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 06:37:20.324328 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":265,"threads_started":1}
I20260619 06:37:20.326148 11991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:20.411873 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 11867
I20260619 06:37:20.429546 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 11999
I20260619 06:37:20.447870 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:20.430579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.28.104.195:45871)
0619 06:37:20.431243 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:20.431257 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:20.431300 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:37:20.432995 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:20.433006 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:20.433561 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:20.433932 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:20.435433 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:20.436668 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:20.438023 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:20.438703 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:20.438900 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:20.441671 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:20.441679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:20.441695 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:20.441731 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:20.444467 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:20.445231 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:20.445246 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:20.445253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:20.445423 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:20.445829 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:20.445833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:20.445835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:20.446370 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:20.446704 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:20.447133 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 06:37:20.447563 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":331,"threads_started":1}
I20260619 06:37:20.449389 12123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:20.533380 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 11999
2026-06-19T06:37:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24351 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:37:20 dist-test-slave-nf4n krb5kdc[12186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:37:20 dist-test-slave-nf4n krb5kdc[12186](info): set up 2 sockets
Jun 19 06:37:20 dist-test-slave-nf4n krb5kdc[12186](info): commencing operation
krb5kdc: starting...
W20260619 06:37:21.641211 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:37:21 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:37:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:37:21.884261 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:40281
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:37:22.217025 12199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:22.217404 12199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:22.217491 12199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:22.224597 12199 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:37:22.224687 12199 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:37:22.224723 12199 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:37:22.224767 12199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:37:22.224822 12199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:22.224887 12199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:22.224931 12199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:37:22.224987 12199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:37:22.234835 12199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:22.236063 12199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:22.237432 12199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:22.248998 12204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:22.249087 12205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:22.249207 12207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:22.249991 12199 server_base.cc:1034] running on GCE node
I20260619 06:37:22.250780 12199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:22.252161 12199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:22.253397 12199 hybrid_clock.cc:648] HybridClock initialized: now 1781851042253334 us; error 40 us; skew 500 ppm
I20260619 06:37:22.253717 12199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:22 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:22.260650 12199 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:37:22.262199 12199 webserver.cc:460] Webserver started at http://127.28.104.254:44005/ using document root <none> and password file <none>
I20260619 06:37:22.262899 12199 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:22.263028 12199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:22.263342 12199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:37:22.266269 12199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "2edcf7f41aa44a1c9ce32011576f2950"
format_stamp: "Formatted at 2026-06-19 06:37:22 on dist-test-slave-nf4n"
server_key: "165d4acafc77c791f94e38496d828f9a"
server_key_iv: "aea1ad70aff81a34e47de2cb7000012d"
server_key_version: "encryptionkey@0"
I20260619 06:37:22.267024 12199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "2edcf7f41aa44a1c9ce32011576f2950"
format_stamp: "Formatted at 2026-06-19 06:37:22 on dist-test-slave-nf4n"
server_key: "165d4acafc77c791f94e38496d828f9a"
server_key_iv: "aea1ad70aff81a34e47de2cb7000012d"
server_key_version: "encryptionkey@0"
I20260619 06:37:22.271981 12199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:37:22.275375 12214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:22.276774 12199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:37:22.276955 12199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "2edcf7f41aa44a1c9ce32011576f2950"
format_stamp: "Formatted at 2026-06-19 06:37:22 on dist-test-slave-nf4n"
server_key: "165d4acafc77c791f94e38496d828f9a"
server_key_iv: "aea1ad70aff81a34e47de2cb7000012d"
server_key_version: "encryptionkey@0"
I20260619 06:37:22.277243 12199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:22.312369 12199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:22.316855 12199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:22.317162 12199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:22.336248 12199 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:40281
I20260619 06:37:22.336333 12265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:40281 every 8 connection(s)
I20260619 06:37:22.337682 12199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:37:22.341756 12266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:37:22.346813 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 12199
I20260619 06:37:22.347106 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:37:22.348055 29091 external_mini_cluster.cc:1428] Setting key 3c7760e0d65dedbbd364126347a8a5b0
I20260619 06:37:22.354933 12266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Bootstrap starting.
I20260619 06:37:22.359043 12266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Neither blocks nor log segments found. Creating new log.
I20260619 06:37:22.360327 12266 log.cc:824] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:22.364077 12266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: No bootstrap required, opened a new log
Jun 19 06:37:22 dist-test-slave-nf4n krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:37:22.372388 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:22.350249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.28.104.254:40281)
0619 06:37:22.351058 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:22.351075 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:22.351116 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:37:22.352962 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:22.352985 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:22.353439 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:22.353769 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:22.355811 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:22.357268 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:22.359394 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:22.360076 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:22.360212 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:22.365148 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:22.365156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:22.365170 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:22.365205 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:22.368310 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:22.369076 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:22.369090 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:22.369097 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:22.369268 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:22.369654 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:22.369660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:22.369672 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:22.370196 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:22.370508 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:22.370977 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 06:37:22.371393 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":391,"threads_started":1}
I20260619 06:37:22.372829 12266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:37:22.373302 12266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:37:22.373421 12266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edcf7f41aa44a1c9ce32011576f2950, State: Initialized, Role: FOLLOWER
I20260619 06:37:22.374004 12266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:37:22.374290 12266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:37:22.374450 12266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:37:22.374655 12266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:37:22.377265 12266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:37:22.377794 12266 leader_election.cc:302] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2edcf7f41aa44a1c9ce32011576f2950; no voters:
I20260619 06:37:22.378372 12266 leader_election.cc:288] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:37:22.378504 12271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:37:22.379231 12271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 1 LEADER]: Becoming Leader. State: Replica: 2edcf7f41aa44a1c9ce32011576f2950, State: Running, Role: LEADER
I20260619 06:37:22.379786 12271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:37:22.380609 12266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:37:22.382666 12273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2edcf7f41aa44a1c9ce32011576f2950. Latest consensus state: current_term: 1 leader_uuid: "2edcf7f41aa44a1c9ce32011576f2950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } } }
I20260619 06:37:22.383129 12273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: This master's current role is: LEADER
I20260619 06:37:22.382848 12272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2edcf7f41aa44a1c9ce32011576f2950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } } }
I20260619 06:37:22.383344 12272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: This master's current role is: LEADER
I20260619 06:37:22.396588 12276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:37:22.402233 12276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:37:22.411520 12276 catalog_manager.cc:1347] Generated new cluster ID: 33217898ca564c43969c38dce61ad3e8
I20260619 06:37:22.411649 12276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:37:22.445446 12276 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:37:22.446810 12276 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:37:22.459005 12276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Generated new TSK 0
I20260619 06:37:22.459961 12276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:37:22.526403 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40281
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:37:22.857642 12292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:22.857910 12292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:22.858006 12292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:22.865394 12292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:22.865507 12292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:22.865622 12292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:37:22.875478 12292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:22.876700 12292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:22.878018 12292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:22.891111 12297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:22.891165 12300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:22.891191 12298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:22.891995 12292 server_base.cc:1034] running on GCE node
I20260619 06:37:22.892575 12292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:22.893573 12292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:22.894826 12292 hybrid_clock.cc:648] HybridClock initialized: now 1781851042894781 us; error 44 us; skew 500 ppm
I20260619 06:37:22.895186 12292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:22 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:22.898610 12292 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:37:22.900102 12292 webserver.cc:460] Webserver started at http://127.28.104.193:37811/ using document root <none> and password file <none>
I20260619 06:37:22.900769 12292 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:22.900872 12292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:22.901203 12292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:37:22.903971 12292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "53f78767d4fa420eb3d659a5c39c68a8"
format_stamp: "Formatted at 2026-06-19 06:37:22 on dist-test-slave-nf4n"
server_key: "f6ad9a33faf7bfec1e71ba78422e49b7"
server_key_iv: "49d4fc2605b731f23338ce4a47eb462a"
server_key_version: "encryptionkey@0"
I20260619 06:37:22.904647 12292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "53f78767d4fa420eb3d659a5c39c68a8"
format_stamp: "Formatted at 2026-06-19 06:37:22 on dist-test-slave-nf4n"
server_key: "f6ad9a33faf7bfec1e71ba78422e49b7"
server_key_iv: "49d4fc2605b731f23338ce4a47eb462a"
server_key_version: "encryptionkey@0"
I20260619 06:37:22.909593 12292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:37:22.912946 12307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:22.914345 12292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:37:22.914469 12292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "53f78767d4fa420eb3d659a5c39c68a8"
format_stamp: "Formatted at 2026-06-19 06:37:22 on dist-test-slave-nf4n"
server_key: "f6ad9a33faf7bfec1e71ba78422e49b7"
server_key_iv: "49d4fc2605b731f23338ce4a47eb462a"
server_key_version: "encryptionkey@0"
I20260619 06:37:22.914628 12292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:22.938063 12292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:22.942312 12292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:22.942560 12292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:22.943581 12292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:22.944918 12292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:22.944999 12292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:22.945150 12292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:22.945197 12292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:22.980222 12292 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:45235
I20260619 06:37:22.980247 12420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:45235 every 8 connection(s)
I20260619 06:37:22.981922 12292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:37:22.989357 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 12292
I20260619 06:37:22.989620 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:37:22.990538 29091 external_mini_cluster.cc:1428] Setting key dc87b019d0dd95c6345b90526804639d
Jun 19 06:37:22 dist-test-slave-nf4n krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:37:23.005915 12423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:22.984682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:55881 (local address 127.28.104.254:40281)
0619 06:37:22.985222 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:22.985229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:22.986187 (+ 958us) server_negotiation.cc:407] Connection header received
0619 06:37:22.988121 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:22.988125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:22.988195 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:22.988294 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:22.990711 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:22.991576 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:22.993757 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:22.994653 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:22.994852 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:22.998932 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:22.998939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:22.998944 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:22.998991 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:23.002146 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:23.003077 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:23.003086 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:23.003089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:23.003261 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:23.003771 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:23.003779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:23.003785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:23.004221 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:23.004373 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:23.005145 (+ 772us) server_negotiation.cc:299] Negotiation successful
0619 06:37:23.005400 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":266,"threads_started":1}
I20260619 06:37:23.006673 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:22.985202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:55881)
0619 06:37:22.986024 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:22.986068 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:22.987845 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:22.988522 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:22.988530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:22.989490 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:37:22.990440 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:22.990471 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:22.991793 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:22.991798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:22.993586 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:22.993599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:22.994900 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:22.994905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:22.995115 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:22.995883 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:37:22.995904 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:37:22.998632 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:37:23.002320 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:23.002334 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:23.002357 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:23.002917 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:23.003384 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:23.003388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:23.003391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:23.003635 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:23.004374 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:37:23.004381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:37:23.004724 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 06:37:23.004963 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 06:37:23.005452 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":292,"threads_started":1}
I20260619 06:37:23.008261 12421 heartbeater.cc:347] Connected to a master server at 127.28.104.254:40281
I20260619 06:37:23.008589 12421 heartbeater.cc:464] Registering TS with master...
I20260619 06:37:23.009500 12421 heartbeater.cc:511] Master 127.28.104.254:40281 requested a full tablet report, sending...
I20260619 06:37:23.011704 12231 ts_manager.cc:194] Registered new tserver with Master: 53f78767d4fa420eb3d659a5c39c68a8 (127.28.104.193:45235)
I20260619 06:37:23.013252 12231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:55881
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:37:23.047703 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40281
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:37:23.373194 12427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:23.373502 12427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:23.373679 12427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:23.380944 12427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:23.381106 12427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:23.381310 12427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:37:23.390724 12427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:23.392019 12427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:23.393412 12427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:23.406641 12433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:23.406575 12432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:23.406598 12435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:23.410362 12427 server_base.cc:1034] running on GCE node
I20260619 06:37:23.410964 12427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:23.411878 12427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:23.413132 12427 hybrid_clock.cc:648] HybridClock initialized: now 1781851043413089 us; error 52 us; skew 500 ppm
I20260619 06:37:23.413445 12427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:23 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851043, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:23.416435 12427 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:37:23.418018 12427 webserver.cc:460] Webserver started at http://127.28.104.194:34743/ using document root <none> and password file <none>
I20260619 06:37:23.418867 12427 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:23.419008 12427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:23.419279 12427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:37:23.422187 12427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "83bb1aa30061484cadc699622de16315"
format_stamp: "Formatted at 2026-06-19 06:37:23 on dist-test-slave-nf4n"
server_key: "33e59c0cccd333f02db09948c292e9e0"
server_key_iv: "ca46bde28831ee74eae75c39fa51c9aa"
server_key_version: "encryptionkey@0"
I20260619 06:37:23.422928 12427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "83bb1aa30061484cadc699622de16315"
format_stamp: "Formatted at 2026-06-19 06:37:23 on dist-test-slave-nf4n"
server_key: "33e59c0cccd333f02db09948c292e9e0"
server_key_iv: "ca46bde28831ee74eae75c39fa51c9aa"
server_key_version: "encryptionkey@0"
I20260619 06:37:23.427963 12427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:37:23.431277 12442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:23.432456 12427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:37:23.432633 12427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "83bb1aa30061484cadc699622de16315"
format_stamp: "Formatted at 2026-06-19 06:37:23 on dist-test-slave-nf4n"
server_key: "33e59c0cccd333f02db09948c292e9e0"
server_key_iv: "ca46bde28831ee74eae75c39fa51c9aa"
server_key_version: "encryptionkey@0"
I20260619 06:37:23.432852 12427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:23.450335 12427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:23.454378 12427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:23.454674 12427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:23.455646 12427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:23.456974 12427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:23.457124 12427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 06:37:23.457267 12427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:23.457340 12427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:23.490696 12427 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:43813
I20260619 06:37:23.490751 12555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:43813 every 8 connection(s)
I20260619 06:37:23.491986 12427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:37:23.500810 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 12427
I20260619 06:37:23.501047 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:37:23.501609 29091 external_mini_cluster.cc:1428] Setting key 19cfb626e6f919da079ab362e8b8c3ca
Jun 19 06:37:23 dist-test-slave-nf4n krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851043, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:37:23.512480 12423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:23.494469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:50621 (local address 127.28.104.254:40281)
0619 06:37:23.494643 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:23.494647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:23.495815 (+ 1168us) server_negotiation.cc:407] Connection header received
0619 06:37:23.497713 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:23.497718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:23.497809 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:23.497915 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:23.499320 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:23.500069 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:23.502393 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:23.502932 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:23.503053 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:23.506864 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:23.506869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:23.506873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:23.506903 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:23.509561 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:23.510268 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:23.510280 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:23.510283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:23.510400 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:23.510818 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:23.510823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:23.510826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:23.511085 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:23.511236 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:23.511904 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 06:37:23.512134 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:37:23.513489 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:23.494968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:50621)
0619 06:37:23.495657 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:23.495695 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:23.497439 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:23.498139 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:23.498147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:23.498625 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:37:23.499162 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:23.499178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:23.500224 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:23.500228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:23.502254 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:23.502266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:23.503131 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:23.503136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:23.503327 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:23.504068 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:37:23.504104 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:37:23.506648 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:37:23.509704 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:23.509713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:23.509732 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:23.510145 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:23.510516 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:23.510522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:23.510525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:23.510704 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:23.511222 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:37:23.511230 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:37:23.511549 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 06:37:23.511806 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 06:37:23.512321 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":263,"threads_started":1}
I20260619 06:37:23.514810 12556 heartbeater.cc:347] Connected to a master server at 127.28.104.254:40281
I20260619 06:37:23.515127 12556 heartbeater.cc:464] Registering TS with master...
I20260619 06:37:23.515929 12556 heartbeater.cc:511] Master 127.28.104.254:40281 requested a full tablet report, sending...
I20260619 06:37:23.517524 12231 ts_manager.cc:194] Registered new tserver with Master: 83bb1aa30061484cadc699622de16315 (127.28.104.194:43813)
I20260619 06:37:23.518750 12231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:50621
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:37:23.554111 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40281
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:37:23.873562 12561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:23.873819 12561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:23.873912 12561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:23.881134 12561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:23.881233 12561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:23.881405 12561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:37:23.890913 12561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:23.892155 12561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:23.893788 12561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:23.906989 12567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:23.907032 12566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:23.906951 12569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:23.913146 12561 server_base.cc:1034] running on GCE node
I20260619 06:37:23.913823 12561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:23.914889 12561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:23.916136 12561 hybrid_clock.cc:648] HybridClock initialized: now 1781851043916068 us; error 55 us; skew 500 ppm
I20260619 06:37:23.916488 12561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:23 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851043, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:23.920286 12561 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:37:23.921757 12561 webserver.cc:460] Webserver started at http://127.28.104.195:43683/ using document root <none> and password file <none>
I20260619 06:37:23.922410 12561 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:23.922505 12561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:23.922780 12561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:37:23.925640 12561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "5986164687a8441f8fbd96cd3e881a34"
format_stamp: "Formatted at 2026-06-19 06:37:23 on dist-test-slave-nf4n"
server_key: "27d82cafc8bf57f8c8bca5ce47faea72"
server_key_iv: "26ef92f102357a9b414111161e1e45b1"
server_key_version: "encryptionkey@0"
I20260619 06:37:23.926370 12561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "5986164687a8441f8fbd96cd3e881a34"
format_stamp: "Formatted at 2026-06-19 06:37:23 on dist-test-slave-nf4n"
server_key: "27d82cafc8bf57f8c8bca5ce47faea72"
server_key_iv: "26ef92f102357a9b414111161e1e45b1"
server_key_version: "encryptionkey@0"
I20260619 06:37:23.931296 12561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:37:23.934587 12576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:23.935992 12561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:37:23.936112 12561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "5986164687a8441f8fbd96cd3e881a34"
format_stamp: "Formatted at 2026-06-19 06:37:23 on dist-test-slave-nf4n"
server_key: "27d82cafc8bf57f8c8bca5ce47faea72"
server_key_iv: "26ef92f102357a9b414111161e1e45b1"
server_key_version: "encryptionkey@0"
I20260619 06:37:23.936328 12561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:23.972501 12561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:23.977012 12561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:23.977298 12561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:23.978215 12561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:23.979481 12561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:23.979604 12561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:23.979753 12561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:23.979831 12561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:24.016026 12561 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:46317
I20260619 06:37:24.016113 12689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:46317 every 8 connection(s)
I20260619 06:37:24.017153 12421 heartbeater.cc:503] Master 127.28.104.254:40281 was elected leader, sending a full tablet report...
I20260619 06:37:24.017508 12561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:37:24.026894 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 12561
I20260619 06:37:24.027143 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:37:24.027612 29091 external_mini_cluster.cc:1428] Setting key 0df20685e2957dd2e2968fe46dd0c058
Jun 19 06:37:24 dist-test-slave-nf4n krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851043, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:37:24.037571 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.020548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:44265 (local address 127.28.104.254:40281)
0619 06:37:24.021096 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.021103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.022090 (+ 987us) server_negotiation.cc:407] Connection header received
0619 06:37:24.023833 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.023838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.023910 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.024016 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:24.025486 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.026286 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.027915 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.028419 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.028545 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.032092 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:24.032097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:24.032100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:24.032131 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:24.034541 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:24.035215 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:24.035220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:24.035222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:24.035334 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:24.035762 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:24.035767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:24.035770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:24.036088 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:24.036238 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.036904 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.037153 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":211,"threads_started":1}
I20260619 06:37:24.038475 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.021272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:44265)
0619 06:37:24.021943 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:24.021980 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:24.023584 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:24.024224 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:24.024231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:24.024711 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:37:24.025319 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.025342 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.026410 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.026413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.027775 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.027787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.028662 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.028669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.028907 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.029681 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:37:24.029702 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:37:24.031887 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:37:24.034672 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:24.034680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:24.034695 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:24.035089 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:24.035442 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:24.035447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:24.035450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:24.035642 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:24.036218 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:37:24.036226 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:37:24.036563 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 06:37:24.036815 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 06:37:24.037358 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":280,"threads_started":1}
I20260619 06:37:24.039786 12690 heartbeater.cc:347] Connected to a master server at 127.28.104.254:40281
I20260619 06:37:24.040089 12690 heartbeater.cc:464] Registering TS with master...
I20260619 06:37:24.040752 12690 heartbeater.cc:511] Master 127.28.104.254:40281 requested a full tablet report, sending...
I20260619 06:37:24.042358 12230 ts_manager.cc:194] Registered new tserver with Master: 5986164687a8441f8fbd96cd3e881a34 (127.28.104.195:46317)
I20260619 06:37:24.043354 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:37:24.043601 12230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:44265
I20260619 06:37:24.060545 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.048563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.28.104.254:40281)
0619 06:37:24.048714 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.048718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.049206 (+ 488us) server_negotiation.cc:407] Connection header received
0619 06:37:24.049346 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.049350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.049405 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.049484 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:24.050906 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.051639 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.053098 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.053521 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.053611 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.055864 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:24.055868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:24.055872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:24.055899 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:24.058345 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:24.058899 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:24.058902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:24.058904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:24.059055 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:24.059365 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:24.059369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:24.059370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:24.059652 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:24.059826 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.060151 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.060320 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:37:24.064570 12230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:37:24.071398 12230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:37:24.096081 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.083741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:46317 (local address 127.0.0.1:46594)
0619 06:37:24.085466 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:24.085520 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:24.085680 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:24.087249 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:24.087256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:24.087284 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:24.087680 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.087696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.089929 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.089935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.092895 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.092910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.095068 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.095075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.095261 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.095316 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 06:37:24.095454 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 06:37:24.095646 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"mutex_wait_us":437,"thread_start_us":1374,"threads_started":1}
I20260619 06:37:24.096879 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.084502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.28.104.194:43813)
0619 06:37:24.086106 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.086114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.086784 (+ 670us) server_negotiation.cc:407] Connection header received
0619 06:37:24.087065 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.087073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.087325 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.087491 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:24.088787 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.090254 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.094471 (+ 4217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.095505 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.095666 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.095823 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.096385 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.096584 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1423,"thread_start_us":1991,"threads_started":1}
I20260619 06:37:24.097299 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.084027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.28.104.195:46317)
0619 06:37:24.086264 (+ 2237us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.086271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.086305 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:37:24.086414 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.086426 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.086694 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.086880 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:24.088091 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.089750 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.093097 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.094542 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.094752 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.094938 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.095535 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.095712 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2080,"thread_start_us":2091,"threads_started":1}
I20260619 06:37:24.097481 12704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.083980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:43813 (local address 127.0.0.1:39438)
0619 06:37:24.086671 (+ 2691us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:24.086692 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:24.086885 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:24.088106 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:24.088110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:24.088137 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:24.088592 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.088605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.090429 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.090435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.092694 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.092710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.095975 (+ 3265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.095981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.096166 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.096205 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:37:24.096925 (+ 720us) client_negotiation.cc:240] Negotiation successful
0619 06:37:24.097081 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2570,"mutex_wait_us":4,"thread_start_us":2740,"threads_started":1}
I20260619 06:37:24.100894 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.085971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.0.0.1:37106)
0619 06:37:24.092874 (+ 6903us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:24.092897 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:24.093008 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:24.094096 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:24.094099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:24.094121 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:24.094470 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.094481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.096324 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.096328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.098482 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.098497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.100007 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.100013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.100224 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.100257 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:37:24.100349 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:37:24.100476 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6755,"thread_start_us":6577,"threads_started":1}
I20260619 06:37:24.101764 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.086302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.28.104.193:45235)
0619 06:37:24.093578 (+ 7276us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.093586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.093612 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:37:24.093704 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.093711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.093910 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.094124 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:24.094653 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.096168 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.098643 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.099802 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.100010 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.100178 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.101293 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.101467 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7107,"thread_start_us":6974,"threads_started":1}
I20260619 06:37:24.101758 12624 tablet_service.cc:1467] Processing CreateTablet for tablet 0e340a03f53342db91ffc78d9331c0cc (DEFAULT_TABLE table=test-table [id=6b51e52e6a074b0cbd297b8815bff37c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:37:24.102294 12490 tablet_service.cc:1467] Processing CreateTablet for tablet 0e340a03f53342db91ffc78d9331c0cc (DEFAULT_TABLE table=test-table [id=6b51e52e6a074b0cbd297b8815bff37c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:37:24.103291 12624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e340a03f53342db91ffc78d9331c0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:37:24.104672 12355 tablet_service.cc:1467] Processing CreateTablet for tablet 0e340a03f53342db91ffc78d9331c0cc (DEFAULT_TABLE table=test-table [id=6b51e52e6a074b0cbd297b8815bff37c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:37:24.105885 12355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e340a03f53342db91ffc78d9331c0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:37:24.105885 12490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e340a03f53342db91ffc78d9331c0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:37:24.114415 12709 tablet_bootstrap.cc:492] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: Bootstrap starting.
I20260619 06:37:24.117530 12709 tablet_bootstrap.cc:654] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: Neither blocks nor log segments found. Creating new log.
I20260619 06:37:24.118682 12709 log.cc:824] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:24.123198 12711 tablet_bootstrap.cc:492] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: Bootstrap starting.
I20260619 06:37:24.126253 12712 tablet_bootstrap.cc:492] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: Bootstrap starting.
I20260619 06:37:24.127269 12711 tablet_bootstrap.cc:654] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: Neither blocks nor log segments found. Creating new log.
I20260619 06:37:24.128281 12709 tablet_bootstrap.cc:492] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: No bootstrap required, opened a new log
I20260619 06:37:24.128367 12711 log.cc:824] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:24.128603 12709 ts_tablet_manager.cc:1397] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.005s
I20260619 06:37:24.129998 12712 tablet_bootstrap.cc:654] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: Neither blocks nor log segments found. Creating new log.
I20260619 06:37:24.131193 12712 log.cc:824] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:24.131717 12711 tablet_bootstrap.cc:492] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: No bootstrap required, opened a new log
I20260619 06:37:24.132045 12711 ts_tablet_manager.cc:1397] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260619 06:37:24.134148 12712 tablet_bootstrap.cc:492] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: No bootstrap required, opened a new log
I20260619 06:37:24.134397 12712 ts_tablet_manager.cc:1397] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 06:37:24.139220 12709 raft_consensus.cc:348] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } }
I20260619 06:37:24.139320 12711 raft_consensus.cc:348] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } }
I20260619 06:37:24.139688 12709 raft_consensus.cc:374] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:37:24.139802 12709 raft_consensus.cc:729] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5986164687a8441f8fbd96cd3e881a34, State: Initialized, Role: FOLLOWER
I20260619 06:37:24.139813 12711 raft_consensus.cc:374] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:37:24.139909 12711 raft_consensus.cc:729] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f78767d4fa420eb3d659a5c39c68a8, State: Initialized, Role: FOLLOWER
I20260619 06:37:24.140487 12709 consensus_queue.cc:260] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } }
I20260619 06:37:24.140839 12711 consensus_queue.cc:260] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } }
I20260619 06:37:24.141646 12690 heartbeater.cc:503] Master 127.28.104.254:40281 was elected leader, sending a full tablet report...
I20260619 06:37:24.142354 12709 ts_tablet_manager.cc:1428] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260619 06:37:24.144189 12711 ts_tablet_manager.cc:1428] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 06:37:24.145989 12712 raft_consensus.cc:348] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } }
I20260619 06:37:24.146431 12712 raft_consensus.cc:374] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:37:24.146546 12712 raft_consensus.cc:729] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bb1aa30061484cadc699622de16315, State: Initialized, Role: FOLLOWER
I20260619 06:37:24.147188 12712 consensus_queue.cc:260] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } }
I20260619 06:37:24.148283 12556 heartbeater.cc:503] Master 127.28.104.254:40281 was elected leader, sending a full tablet report...
I20260619 06:37:24.148947 12712 ts_tablet_manager.cc:1428] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260619 06:37:24.214481 12717 raft_consensus.cc:482] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:37:24.214805 12717 raft_consensus.cc:504] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } }
I20260619 06:37:24.216521 12717 leader_election.cc:288] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5986164687a8441f8fbd96cd3e881a34 (127.28.104.195:46317), 53f78767d4fa420eb3d659a5c39c68a8 (127.28.104.193:45235)
I20260619 06:37:24.223829 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.217165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:36597 (local address 127.28.104.193:45235)
0619 06:37:24.217383 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.217388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.218107 (+ 719us) server_negotiation.cc:407] Connection header received
0619 06:37:24.218274 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.218278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.218336 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.218448 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:24.219275 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.220107 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.222047 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.222820 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.223001 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.223085 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.223442 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.223578 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 06:37:24.223836 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.216821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:57957 (local address 127.28.104.195:46317)
0619 06:37:24.217056 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.217063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.217841 (+ 778us) server_negotiation.cc:407] Connection header received
0619 06:37:24.218201 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.218210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.218287 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.218393 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:24.219165 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.220114 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.222054 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.222811 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.222973 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.223063 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.223443 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.223576 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:37:24.223980 12719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.217048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.28.104.194:36597)
0619 06:37:24.217973 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:24.217991 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:24.218139 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:24.218476 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:24.218481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:24.218506 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:24.219107 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.219119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.220227 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.220235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.221894 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.221906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.223098 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.223106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.223255 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.223310 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 06:37:24.223413 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:37:24.223542 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":81,"thread_start_us":545,"threads_started":1}
I20260619 06:37:24.224324 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.216690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:46317 (local address 127.28.104.194:57957)
0619 06:37:24.217697 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:24.217719 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:24.217992 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:24.218436 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:24.218442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:24.218467 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:24.219010 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.219032 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.220247 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.220252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.221896 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:24.221908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.223068 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.223077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:24.223245 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.223310 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 06:37:24.223413 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:37:24.223550 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":76,"thread_start_us":576,"threads_started":1}
I20260619 06:37:24.224973 12375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e340a03f53342db91ffc78d9331c0cc" candidate_uuid: "83bb1aa30061484cadc699622de16315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f78767d4fa420eb3d659a5c39c68a8" is_pre_election: true
I20260619 06:37:24.225281 12645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e340a03f53342db91ffc78d9331c0cc" candidate_uuid: "83bb1aa30061484cadc699622de16315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5986164687a8441f8fbd96cd3e881a34" is_pre_election: true
I20260619 06:37:24.225555 12375 raft_consensus.cc:2454] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83bb1aa30061484cadc699622de16315 in term 0.
I20260619 06:37:24.225790 12645 raft_consensus.cc:2454] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83bb1aa30061484cadc699622de16315 in term 0.
I20260619 06:37:24.226286 12443 leader_election.cc:302] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f78767d4fa420eb3d659a5c39c68a8, 83bb1aa30061484cadc699622de16315; no voters:
I20260619 06:37:24.226713 12717 raft_consensus.cc:2781] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:37:24.226830 12717 raft_consensus.cc:482] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:37:24.226943 12717 raft_consensus.cc:3037] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:37:24.229140 12717 raft_consensus.cc:504] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } }
I20260619 06:37:24.229790 12717 leader_election.cc:288] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [CANDIDATE]: Term 1 election: Requested vote from peers 5986164687a8441f8fbd96cd3e881a34 (127.28.104.195:46317), 53f78767d4fa420eb3d659a5c39c68a8 (127.28.104.193:45235)
I20260619 06:37:24.230206 12645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e340a03f53342db91ffc78d9331c0cc" candidate_uuid: "83bb1aa30061484cadc699622de16315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5986164687a8441f8fbd96cd3e881a34"
I20260619 06:37:24.230418 12645 raft_consensus.cc:3037] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:37:24.230444 12375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e340a03f53342db91ffc78d9331c0cc" candidate_uuid: "83bb1aa30061484cadc699622de16315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f78767d4fa420eb3d659a5c39c68a8"
I20260619 06:37:24.230687 12375 raft_consensus.cc:3037] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:37:24.232573 12645 raft_consensus.cc:2454] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83bb1aa30061484cadc699622de16315 in term 1.
I20260619 06:37:24.232923 12375 raft_consensus.cc:2454] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83bb1aa30061484cadc699622de16315 in term 1.
I20260619 06:37:24.232947 12444 leader_election.cc:302] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5986164687a8441f8fbd96cd3e881a34, 83bb1aa30061484cadc699622de16315; no voters:
I20260619 06:37:24.233314 12717 raft_consensus.cc:2781] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:37:24.233753 12717 raft_consensus.cc:686] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 1 LEADER]: Becoming Leader. State: Replica: 83bb1aa30061484cadc699622de16315, State: Running, Role: LEADER
W20260619 06:37:24.234246 12422 tablet.cc:2369] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:37:24.234200 12717 consensus_queue.cc:237] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } }
I20260619 06:37:24.239015 12230 catalog_manager.cc:5495] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83bb1aa30061484cadc699622de16315 (127.28.104.194). New cstate: current_term: 1 leader_uuid: "83bb1aa30061484cadc699622de16315" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } health_report { overall_health: UNKNOWN } } }
W20260619 06:37:24.244062 12557 tablet.cc:2369] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:37:24.254431 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.248975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.28.104.194:43813)
0619 06:37:24.249189 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.249195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.249309 (+ 114us) server_negotiation.cc:407] Connection header received
0619 06:37:24.249547 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.249551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.249633 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.249797 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:37:24.250174 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.250873 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.252299 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.252708 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.252838 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.253385 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:37:24.253867 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:37:24.254030 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.254082 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.254158 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:37:24.261976 12375 raft_consensus.cc:1261] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8 [term 1 FOLLOWER]: Refusing update from remote peer 83bb1aa30061484cadc699622de16315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:37:24.262017 12645 raft_consensus.cc:1261] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34 [term 1 FOLLOWER]: Refusing update from remote peer 83bb1aa30061484cadc699622de16315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:37:24.262967 12720 consensus_queue.cc:1035] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:37:24.263274 12717 consensus_queue.cc:1035] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:37:24.269627 12691 tablet.cc:2369] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:37:24.276005 12725 mvcc.cc:205] Tried to move back new op lower bound from 7298461877287809024 to 7298461877187059712. Current Snapshot: MvccSnapshot[applied={T|T < 7298461877287809024}]
I20260619 06:37:24.290081 12230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:37:24.290335 12230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:37:24.293797 12230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Sending DeleteTablet for 3 replicas of tablet 0e340a03f53342db91ffc78d9331c0cc
I20260619 06:37:24.295128 12624 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e340a03f53342db91ffc78d9331c0cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:37:24 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:46594
I20260619 06:37:24.295132 12355 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e340a03f53342db91ffc78d9331c0cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:37:24 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:37106
I20260619 06:37:24.295671 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 12292
I20260619 06:37:24.295997 12490 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e340a03f53342db91ffc78d9331c0cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:37:24 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:39438
I20260619 06:37:24.301282 12735 tablet_replica.cc:330] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: stopping tablet replica
I20260619 06:37:24.301890 12735 raft_consensus.cc:2229] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:37:24.309369 12732 tablet_replica.cc:330] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: stopping tablet replica
I20260619 06:37:24.310175 12732 raft_consensus.cc:2229] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:37:24.310977 12732 raft_consensus.cc:2258] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:37:24.312211 12733 tablet_replica.cc:330] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: stopping tablet replica
I20260619 06:37:24.312743 12733 raft_consensus.cc:2229] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:37:24.313355 12733 raft_consensus.cc:2258] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:37:24.313619 12735 raft_consensus.cc:2258] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:37:24.313835 12732 ts_tablet_manager.cc:1905] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:24.316049 12733 ts_tablet_manager.cc:1905] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 06:37:24 dist-test-slave-nf4n krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:37:24.321133 12732 ts_tablet_manager.cc:1918] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:24.321355 12732 log.cc:1196] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/0e340a03f53342db91ffc78d9331c0cc
I20260619 06:37:24.321972 12732 ts_tablet_manager.cc:1939] T 0e340a03f53342db91ffc78d9331c0cc P 5986164687a8441f8fbd96cd3e881a34: Deleting consensus metadata
I20260619 06:37:24.324040 12216 catalog_manager.cc:4841] TS 5986164687a8441f8fbd96cd3e881a34 (127.28.104.195:46317): tablet 0e340a03f53342db91ffc78d9331c0cc (table test-table [id=6b51e52e6a074b0cbd297b8815bff37c]) successfully deleted
I20260619 06:37:24.324769 12733 ts_tablet_manager.cc:1918] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:24.324929 12733 log.cc:1196] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/0e340a03f53342db91ffc78d9331c0cc
I20260619 06:37:24.325446 12733 ts_tablet_manager.cc:1939] T 0e340a03f53342db91ffc78d9331c0cc P 53f78767d4fa420eb3d659a5c39c68a8: Deleting consensus metadata
I20260619 06:37:24.327250 12215 catalog_manager.cc:4841] TS 53f78767d4fa420eb3d659a5c39c68a8 (127.28.104.193:45235): tablet 0e340a03f53342db91ffc78d9331c0cc (table test-table [id=6b51e52e6a074b0cbd297b8815bff37c]) successfully deleted
I20260619 06:37:24.328545 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.296751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.28.104.193:45235)
0619 06:37:24.296976 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.296985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.307297 (+ 10312us) server_negotiation.cc:407] Connection header received
0619 06:37:24.307746 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.307752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.307841 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.307922 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:24.310151 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.311374 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.313359 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.314069 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.314194 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.320215 (+ 6021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:24.320227 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:24.320245 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:24.320295 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:24.324278 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:24.325254 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:24.325264 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:24.325268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:24.325485 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:24.326050 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:24.326055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:24.326058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:24.326525 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:24.326829 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.327870 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.328108 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:37:24.329252 12416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:24.329454 12735 ts_tablet_manager.cc:1905] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:24.340648 12735 ts_tablet_manager.cc:1918] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:24.341071 12735 log.cc:1196] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/0e340a03f53342db91ffc78d9331c0cc
I20260619 06:37:24.341848 12735 ts_tablet_manager.cc:1939] T 0e340a03f53342db91ffc78d9331c0cc P 83bb1aa30061484cadc699622de16315: Deleting consensus metadata
I20260619 06:37:24.344246 12218 catalog_manager.cc:4841] TS 83bb1aa30061484cadc699622de16315 (127.28.104.194:43813): tablet 0e340a03f53342db91ffc78d9331c0cc (table test-table [id=6b51e52e6a074b0cbd297b8815bff37c]) successfully deleted
I20260619 06:37:24.440196 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 12292
I20260619 06:37:24.459120 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 12427
Jun 19 06:37:24 dist-test-slave-nf4n krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:37:24.476341 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.459888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.28.104.194:43813)
0619 06:37:24.460084 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.460090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.460111 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:37:24.460261 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.460267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.460361 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.460492 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:24.461887 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.463139 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.464524 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.465268 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.465407 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.469959 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:24.469974 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:24.469990 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:24.470030 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:24.473541 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:24.474387 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:24.474394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:24.474397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:24.474515 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:24.474951 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:24.474954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:24.474956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:24.475295 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:24.475556 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.475859 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.476017 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:37:24.477128 12551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:24.592660 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 12427
I20260619 06:37:24.611169 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 12561
Jun 19 06:37:24 dist-test-slave-nf4n krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:37:24.627859 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:24.612013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.28.104.195:46317)
0619 06:37:24.612201 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:24.612208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:24.612255 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:37:24.612486 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:24.612491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:24.612567 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:24.612684 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:24.614130 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.614848 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.616752 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:24.617365 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:24.617495 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:24.622081 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:24.622094 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:24.622111 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:24.622144 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:24.624943 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:24.625783 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:24.625792 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:24.625796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:24.625938 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:24.626308 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:24.626315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:24.626318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:24.626675 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:24.626997 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:24.627310 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 06:37:24.627536 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:37:24.628580 12685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:24.717098 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 12561
I20260619 06:37:24.734323 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 12199
I20260619 06:37:24.735560 12261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:24.851006 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 12199
I20260619 06:37:44.872673 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:40281
--webserver_interface=127.28.104.254
--webserver_port=44005
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:37:45.206544 12740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:45.206885 12740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:45.207053 12740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:45.214447 12740 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:37:45.214581 12740 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:37:45.214675 12740 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:37:45.214769 12740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:37:45.214870 12740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:45.214972 12740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:45.215056 12740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:37:45.215137 12740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:37:45.225261 12740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=44005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:45.226598 12740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:45.227954 12740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:45.250597 12746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:45.250590 12747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:45.251149 12749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:45.251570 12740 server_base.cc:1034] running on GCE node
I20260619 06:37:45.252462 12740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:45.254128 12740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:45.255411 12740 hybrid_clock.cc:648] HybridClock initialized: now 1781851065255338 us; error 44 us; skew 500 ppm
I20260619 06:37:45.255793 12740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:45 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:45.264278 12740 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:37:45.266098 12740 webserver.cc:460] Webserver started at http://127.28.104.254:44005/ using document root <none> and password file <none>
I20260619 06:37:45.266916 12740 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:45.267078 12740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:45.273466 12740 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 06:37:45.276317 12756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:45.277840 12740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:37:45.278028 12740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "2edcf7f41aa44a1c9ce32011576f2950"
format_stamp: "Formatted at 2026-06-19 06:37:22 on dist-test-slave-nf4n"
server_key: "165d4acafc77c791f94e38496d828f9a"
server_key_iv: "aea1ad70aff81a34e47de2cb7000012d"
server_key_version: "encryptionkey@0"
I20260619 06:37:45.279080 12740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:45.298023 12740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:45.302317 12740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:45.302618 12740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:45.322705 12740 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:40281
I20260619 06:37:45.322731 12808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:40281 every 8 connection(s)
I20260619 06:37:45.324286 12740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:37:45.326519 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 12740
I20260619 06:37:45.327637 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:45235
--local_ip_for_outbound_sockets=127.28.104.193
--tserver_master_addrs=127.28.104.254:40281
--webserver_port=37811
--webserver_interface=127.28.104.193
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:37:45.335690 12809 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:37:45.340075 12809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Bootstrap starting.
I20260619 06:37:45.362185 12809 log.cc:824] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:45.375656 12809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:37:45.376283 12809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Bootstrap complete.
I20260619 06:37:45.387409 12809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:37:45.388339 12809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edcf7f41aa44a1c9ce32011576f2950, State: Initialized, Role: FOLLOWER
I20260619 06:37:45.389130 12809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:37:45.389397 12809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:37:45.389531 12809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:37:45.389737 12809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:37:45.393961 12809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:37:45.394531 12809 leader_election.cc:302] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2edcf7f41aa44a1c9ce32011576f2950; no voters:
I20260619 06:37:45.395246 12809 leader_election.cc:288] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:37:45.395403 12813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:37:45.396808 12813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 2 LEADER]: Becoming Leader. State: Replica: 2edcf7f41aa44a1c9ce32011576f2950, State: Running, Role: LEADER
I20260619 06:37:45.397509 12813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:37:45.397804 12809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:37:45.400106 12815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2edcf7f41aa44a1c9ce32011576f2950. Latest consensus state: current_term: 2 leader_uuid: "2edcf7f41aa44a1c9ce32011576f2950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } } }
I20260619 06:37:45.400461 12815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: This master's current role is: LEADER
I20260619 06:37:45.400480 12814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2edcf7f41aa44a1c9ce32011576f2950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } } }
I20260619 06:37:45.401175 12814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: This master's current role is: LEADER
I20260619 06:37:45.403451 12820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:37:45.412184 12820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:37:45.413688 12820 catalog_manager.cc:1259] Loaded cluster ID: 33217898ca564c43969c38dce61ad3e8
I20260619 06:37:45.413836 12820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:37:45.418488 12820 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:37:45.420145 12820 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Loaded TSK: 0
I20260619 06:37:45.431667 12830 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:37:45.455374 12820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Generated new TSK 1
I20260619 06:37:45.455786 12820 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:37:45.693661 12811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:45.693994 12811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:45.694175 12811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:45.701954 12811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:45.702133 12811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:45.702323 12811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:37:45.712621 12811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:45235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=37811
--tserver_master_addrs=127.28.104.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:45.714095 12811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:45.715524 12811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:45.731850 12837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:45.732224 12839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:45.732800 12836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:45.733901 12811 server_base.cc:1034] running on GCE node
I20260619 06:37:45.734551 12811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:45.735582 12811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:45.736812 12811 hybrid_clock.cc:648] HybridClock initialized: now 1781851065736734 us; error 74 us; skew 500 ppm
I20260619 06:37:45.737208 12811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:45 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:45.740337 12811 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:37:45.742010 12811 webserver.cc:460] Webserver started at http://127.28.104.193:37811/ using document root <none> and password file <none>
I20260619 06:37:45.742694 12811 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:45.742833 12811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:45.748543 12811 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:37:45.751518 12846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:45.753015 12811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:37:45.753242 12811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "53f78767d4fa420eb3d659a5c39c68a8"
format_stamp: "Formatted at 2026-06-19 06:37:22 on dist-test-slave-nf4n"
server_key: "f6ad9a33faf7bfec1e71ba78422e49b7"
server_key_iv: "49d4fc2605b731f23338ce4a47eb462a"
server_key_version: "encryptionkey@0"
I20260619 06:37:45.754333 12811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:45.775645 12811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:45.780392 12811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:45.780678 12811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:45.781818 12811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:45.783275 12811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:45.783403 12811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:45.783541 12811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:45.783622 12811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:45.818122 12811 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:45235
I20260619 06:37:45.818207 12959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:45235 every 8 connection(s)
I20260619 06:37:45.819591 12811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:37:45.825572 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 12811
I20260619 06:37:45.826359 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:43813
--local_ip_for_outbound_sockets=127.28.104.194
--tserver_master_addrs=127.28.104.254:40281
--webserver_port=34743
--webserver_interface=127.28.104.194
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:37:45 dist-test-slave-nf4n krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:37:45.845579 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:45.822606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:47621)
0619 06:37:45.823408 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:45.823458 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:45.825246 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:45.826880 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:45.826887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:45.827381 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:37:45.827919 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:45.827937 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:45.830305 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:45.830308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:45.831532 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:45.831544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:45.833087 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:45.833093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:45.833280 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:45.834036 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:37:45.834064 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:37:45.836413 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:37:45.841126 (+ 4713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:45.841134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:45.841146 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:45.841601 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:45.842124 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:45.842127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:45.842130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:45.842311 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:45.843320 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:37:45.843326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:37:45.843655 (+ 329us) client_negotiation.cc:769] Sending connection context
0619 06:37:45.843892 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 06:37:45.844294 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":344,"threads_started":1}
I20260619 06:37:45.846010 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:45.822427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:47621 (local address 127.28.104.254:40281)
0619 06:37:45.823022 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:45.823035 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:45.823605 (+ 570us) server_negotiation.cc:407] Connection header received
0619 06:37:45.825851 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:45.825871 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:45.826329 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:45.826756 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:45.828152 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:45.830142 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:45.831682 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:45.832820 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:45.833009 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:45.836705 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:45.836718 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:45.836739 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:45.836790 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:45.840905 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:45.841748 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:45.841760 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:45.841771 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:45.841986 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:45.842459 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:45.842476 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:45.842482 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:45.843173 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:45.843367 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:45.844149 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 06:37:45.844603 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":281,"threads_started":1}
I20260619 06:37:45.849275 12960 heartbeater.cc:347] Connected to a master server at 127.28.104.254:40281
I20260619 06:37:45.849599 12960 heartbeater.cc:464] Registering TS with master...
I20260619 06:37:45.850322 12960 heartbeater.cc:511] Master 127.28.104.254:40281 requested a full tablet report, sending...
I20260619 06:37:45.852782 12774 ts_manager.cc:194] Registered new tserver with Master: 53f78767d4fa420eb3d659a5c39c68a8 (127.28.104.193:45235)
I20260619 06:37:45.855119 12774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:47621
W20260619 06:37:46.159000 12964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:46.159255 12964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:46.159346 12964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:46.166472 12964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:46.166579 12964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:46.166687 12964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:37:46.176115 12964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:43813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=34743
--tserver_master_addrs=127.28.104.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:46.177322 12964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:46.178577 12964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:46.191457 12972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:46.191411 12970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:46.191504 12969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:46.192287 12964 server_base.cc:1034] running on GCE node
I20260619 06:37:46.192929 12964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:46.193882 12964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:46.195116 12964 hybrid_clock.cc:648] HybridClock initialized: now 1781851066195070 us; error 37 us; skew 500 ppm
I20260619 06:37:46.195436 12964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:46 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:46.198572 12964 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:37:46.199982 12964 webserver.cc:460] Webserver started at http://127.28.104.194:34743/ using document root <none> and password file <none>
I20260619 06:37:46.200608 12964 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:46.200716 12964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:46.206112 12964 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:37:46.208636 12979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:46.210028 12964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:37:46.210161 12964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "83bb1aa30061484cadc699622de16315"
format_stamp: "Formatted at 2026-06-19 06:37:23 on dist-test-slave-nf4n"
server_key: "33e59c0cccd333f02db09948c292e9e0"
server_key_iv: "ca46bde28831ee74eae75c39fa51c9aa"
server_key_version: "encryptionkey@0"
I20260619 06:37:46.211057 12964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:46.225320 12964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:46.229315 12964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:46.229544 12964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:46.230494 12964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:46.231827 12964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:46.231911 12964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:46.231983 12964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:46.232040 12964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:46.266661 12964 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:43813
I20260619 06:37:46.266747 13092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:43813 every 8 connection(s)
I20260619 06:37:46.268180 12964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:37:46.274418 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 12964
I20260619 06:37:46.275175 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:46317
--local_ip_for_outbound_sockets=127.28.104.195
--tserver_master_addrs=127.28.104.254:40281
--webserver_port=43683
--webserver_interface=127.28.104.195
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:37:46 dist-test-slave-nf4n krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:37:46.291294 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.270682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:37709 (local address 127.28.104.254:40281)
0619 06:37:46.270936 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:46.270943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:46.272033 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 06:37:46.273895 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:46.273900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:46.273987 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:46.274079 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:46.276153 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.276837 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.279333 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.279851 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.279948 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.284258 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:46.284268 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:46.284276 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:46.284323 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:46.287951 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:46.288663 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:46.288671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:46.288678 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:46.288826 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:46.289277 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:46.289284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:46.289290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:46.289723 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:46.289880 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:46.290543 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 06:37:46.290844 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:37:46.291940 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.271202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:37709)
0619 06:37:46.271868 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:46.271914 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:46.273622 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:46.274365 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:46.274377 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:46.275130 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:37:46.275966 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.275994 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.277016 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.277049 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.279177 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.279195 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.280077 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.280086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.280410 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.281165 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:37:46.281199 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:37:46.284020 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:37:46.288099 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:46.288105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:46.288118 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:46.288528 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:46.288929 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:46.288932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:46.288934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:46.289161 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:46.289843 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:37:46.289848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:37:46.290170 (+ 322us) client_negotiation.cc:769] Sending connection context
0619 06:37:46.290390 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 06:37:46.290786 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":256,"threads_started":1}
I20260619 06:37:46.293483 13093 heartbeater.cc:347] Connected to a master server at 127.28.104.254:40281
I20260619 06:37:46.293790 13093 heartbeater.cc:464] Registering TS with master...
I20260619 06:37:46.294489 13093 heartbeater.cc:511] Master 127.28.104.254:40281 requested a full tablet report, sending...
I20260619 06:37:46.295948 12774 ts_manager.cc:194] Registered new tserver with Master: 83bb1aa30061484cadc699622de16315 (127.28.104.194:43813)
I20260619 06:37:46.296736 12774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:37709
W20260619 06:37:46.604668 13096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:37:46.604940 13096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:37:46.605098 13096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:37:46.612277 13096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:37:46.612375 13096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:37:46.612540 13096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:37:46.622277 13096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:46317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=43683
--tserver_master_addrs=127.28.104.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:37:46.623580 13096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:37:46.624900 13096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:37:46.638020 13101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:46.638055 13102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:37:46.637991 13104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:37:46.639283 13096 server_base.cc:1034] running on GCE node
I20260619 06:37:46.639871 13096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:37:46.640767 13096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:37:46.641978 13096 hybrid_clock.cc:648] HybridClock initialized: now 1781851066641911 us; error 50 us; skew 500 ppm
I20260619 06:37:46.642318 13096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:37:46 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:37:46.645521 13096 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:37:46.647126 13096 webserver.cc:460] Webserver started at http://127.28.104.195:43683/ using document root <none> and password file <none>
I20260619 06:37:46.647805 13096 fs_manager.cc:359] Metadata directory not provided
I20260619 06:37:46.647964 13096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:37:46.653254 13096 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:37:46.655810 13111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:46.657162 13096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:37:46.657373 13096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "5986164687a8441f8fbd96cd3e881a34"
format_stamp: "Formatted at 2026-06-19 06:37:23 on dist-test-slave-nf4n"
server_key: "27d82cafc8bf57f8c8bca5ce47faea72"
server_key_iv: "26ef92f102357a9b414111161e1e45b1"
server_key_version: "encryptionkey@0"
I20260619 06:37:46.658329 13096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:37:46.686305 13096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:37:46.690464 13096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:37:46.690745 13096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:37:46.691718 13096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:37:46.693130 13096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:37:46.693254 13096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:46.693387 13096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:37:46.693472 13096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:37:46.727552 13096 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:46317
I20260619 06:37:46.727640 13225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:46317 every 8 connection(s)
I20260619 06:37:46.729384 13096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:37:46.733373 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 13096
Jun 19 06:37:46 dist-test-slave-nf4n krb5kdc[12186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:37:46.750993 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.732015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:55253 (local address 127.28.104.254:40281)
0619 06:37:46.732232 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:46.732238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:46.733602 (+ 1364us) server_negotiation.cc:407] Connection header received
0619 06:37:46.735785 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:46.735791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:46.735884 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:46.735997 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:46.737573 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.738294 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.739847 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.740324 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.740463 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.743995 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:46.744000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:46.744003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:46.744033 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:46.747595 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:46.748602 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:46.748608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:46.748617 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:46.748734 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:46.749228 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:46.749233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:46.749236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:46.749509 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:46.749662 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:46.750361 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 06:37:46.750612 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:37:46.751989 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.734314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.28.104.254:40281)
0619 06:37:46.734786 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:46.734792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:46.737592 (+ 2800us) server_negotiation.cc:407] Connection header received
0619 06:37:46.737794 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:46.737799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:46.737856 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:46.737965 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:46.739479 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.740233 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.741910 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.742330 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.742485 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.745338 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:46.745343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:46.745346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:46.745387 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:46.748573 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:46.749357 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:46.749362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:46.749375 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:46.749495 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:46.749963 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:46.749967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:46.749969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:46.750414 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:46.750774 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:46.751418 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 06:37:46.751655 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":234,"threads_started":1}
I20260619 06:37:46.752101 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.732509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:55253)
0619 06:37:46.733418 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:46.733468 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:46.735514 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:46.736238 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:46.736246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:46.736744 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:37:46.737403 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.737423 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.738474 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.738478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.739684 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.739707 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.740565 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.740573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.740803 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.741580 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:37:46.741602 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:37:46.743762 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:37:46.747780 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:46.747790 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:46.747808 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:46.748269 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:46.748847 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:37:46.748852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:37:46.748856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:37:46.749104 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:37:46.749637 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:37:46.749645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:37:46.749997 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 06:37:46.750270 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 06:37:46.750808 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":324,"threads_started":1}
I20260619 06:37:46.754104 13226 heartbeater.cc:347] Connected to a master server at 127.28.104.254:40281
I20260619 06:37:46.754447 13226 heartbeater.cc:464] Registering TS with master...
I20260619 06:37:46.755219 13226 heartbeater.cc:511] Master 127.28.104.254:40281 requested a full tablet report, sending...
I20260619 06:37:46.756573 12774 ts_manager.cc:194] Registered new tserver with Master: 5986164687a8441f8fbd96cd3e881a34 (127.28.104.195:46317)
I20260619 06:37:46.757542 12774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:55253
I20260619 06:37:46.767290 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:37:46.774881 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.768260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.28.104.254:40281)
0619 06:37:46.768479 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:46.768483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:46.769081 (+ 598us) server_negotiation.cc:407] Connection header received
0619 06:37:46.769350 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:46.769354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:46.769488 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:46.769638 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:37:46.770205 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.770919 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.772522 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.772951 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.773126 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.773628 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:37:46.774056 (+ 428us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 06:37:46.774511 (+ 455us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35578: token expired
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 06:37:46.775110 13229 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35578: token expired
W20260619 06:37:46.775714 12695 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.104.254:40281)
I20260619 06:37:46.794121 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.776325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.28.104.254:40281)
0619 06:37:46.776501 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:46.776509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:46.776638 (+ 129us) server_negotiation.cc:407] Connection header received
0619 06:37:46.776844 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:46.776851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:46.776932 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:46.777080 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:46.779476 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.780652 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.782711 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.783363 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.783491 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.787403 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:46.787412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:46.787419 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:46.787459 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:46.791513 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:46.792215 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:46.792218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:46.792221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:46.792341 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:46.792824 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:46.792826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:46.792829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:46.793214 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:46.793386 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:46.793691 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 06:37:46.793849 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:37:46.797412 12774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:37:46.804682 12774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:37:46.844673 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.816948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:46317 (local address 127.0.0.1:34900)
0619 06:37:46.831550 (+ 14602us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:46.831601 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:46.833327 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:46.834148 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:46.834156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:46.834187 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:46.834579 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.834592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.837887 (+ 3295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.837895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.840467 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.840484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.843224 (+ 2740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.843232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.843554 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.843624 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 06:37:46.843783 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 06:37:46.844090 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14299,"mutex_wait_us":3,"thread_start_us":14588,"threads_started":1}
I20260619 06:37:46.845703 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.823012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.28.104.195:46317)
0619 06:37:46.831362 (+ 8350us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:46.831372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:46.833275 (+ 1903us) server_negotiation.cc:407] Connection header received
0619 06:37:46.833560 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:46.833571 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:46.833915 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:46.834136 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:46.834793 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.837765 (+ 2972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.840746 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.842556 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.842778 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.843030 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:46.845135 (+ 2105us) server_negotiation.cc:299] Negotiation successful
0619 06:37:46.845336 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8067,"thread_start_us":7689,"threads_started":1}
I20260619 06:37:46.847235 13237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.819473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:43813 (local address 127.0.0.1:50550)
0619 06:37:46.835364 (+ 15891us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:46.835382 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:46.835485 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:46.838414 (+ 2929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:46.838419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:46.838445 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:46.839540 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.839553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.841474 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.841479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.843803 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.843817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.846324 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.846328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.846562 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.846595 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:37:46.846683 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:37:46.846824 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15762,"mutex_wait_us":101,"thread_start_us":12905,"threads_started":1}
I20260619 06:37:46.847945 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.820047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.28.104.194:43813)
0619 06:37:46.837529 (+ 17482us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:46.837539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:46.837572 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:37:46.837677 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:46.837686 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:46.837950 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:46.838158 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:46.839739 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.841312 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.844006 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.845802 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.845976 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.846142 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:46.847452 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0619 06:37:46.847630 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17311,"thread_start_us":17182,"threads_started":1}
I20260619 06:37:46.850634 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.817005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.0.0.1:60160)
0619 06:37:46.832661 (+ 15656us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:46.832683 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:46.832868 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:46.838435 (+ 5567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:46.838439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:46.838455 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:46.839005 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.839022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.841480 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.841484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.844105 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.844118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.846315 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.846322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.846661 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.846696 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:37:46.846762 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 06:37:46.850199 (+ 3437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15498,"mutex_wait_us":471,"thread_start_us":17940,"threads_started":1}
I20260619 06:37:46.851619 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.817464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.28.104.193:45235)
0619 06:37:46.837896 (+ 20432us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:46.837905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:46.837931 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:37:46.838026 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:46.838033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:46.838243 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:46.838546 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:46.839187 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.841312 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.844278 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.845802 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.845990 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.846586 (+ 596us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:46.851020 (+ 4434us) server_negotiation.cc:299] Negotiation successful
0619 06:37:46.851240 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20273,"thread_start_us":20148,"threads_started":1}
I20260619 06:37:46.852104 13159 tablet_service.cc:1467] Processing CreateTablet for tablet eb5d4b1c65514cef98f12eaff463d6c2 (DEFAULT_TABLE table=test-table [id=6c12bc07412f4014b986a868e9982f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:37:46.852639 13028 tablet_service.cc:1467] Processing CreateTablet for tablet eb5d4b1c65514cef98f12eaff463d6c2 (DEFAULT_TABLE table=test-table [id=6c12bc07412f4014b986a868e9982f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:37:46.853824 13159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5d4b1c65514cef98f12eaff463d6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:37:46.855273 13028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5d4b1c65514cef98f12eaff463d6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:37:46.856575 12895 tablet_service.cc:1467] Processing CreateTablet for tablet eb5d4b1c65514cef98f12eaff463d6c2 (DEFAULT_TABLE table=test-table [id=6c12bc07412f4014b986a868e9982f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:37:46.858054 12895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5d4b1c65514cef98f12eaff463d6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:37:46.858433 12960 heartbeater.cc:503] Master 127.28.104.254:40281 was elected leader, sending a full tablet report...
I20260619 06:37:46.869724 13242 tablet_bootstrap.cc:492] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: Bootstrap starting.
I20260619 06:37:46.873744 13242 tablet_bootstrap.cc:654] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: Neither blocks nor log segments found. Creating new log.
I20260619 06:37:46.875324 13242 log.cc:824] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:46.880928 13243 tablet_bootstrap.cc:492] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: Bootstrap starting.
I20260619 06:37:46.880934 13244 tablet_bootstrap.cc:492] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Bootstrap starting.
I20260619 06:37:46.884614 13244 tablet_bootstrap.cc:654] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Neither blocks nor log segments found. Creating new log.
I20260619 06:37:46.885934 13244 log.cc:824] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:46.887249 13243 tablet_bootstrap.cc:654] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: Neither blocks nor log segments found. Creating new log.
I20260619 06:37:46.888121 13242 tablet_bootstrap.cc:492] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: No bootstrap required, opened a new log
I20260619 06:37:46.888500 13242 ts_tablet_manager.cc:1397] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.012s
I20260619 06:37:46.888756 13243 log.cc:824] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: Log is configured to *not* fsync() on all Append() calls
I20260619 06:37:46.891885 13243 tablet_bootstrap.cc:492] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: No bootstrap required, opened a new log
I20260619 06:37:46.892139 13243 ts_tablet_manager.cc:1397] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260619 06:37:46.893460 13244 tablet_bootstrap.cc:492] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: No bootstrap required, opened a new log
I20260619 06:37:46.893720 13244 ts_tablet_manager.cc:1397] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20260619 06:37:46.897943 13242 raft_consensus.cc:348] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } }
I20260619 06:37:46.898341 13242 raft_consensus.cc:374] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:37:46.898444 13242 raft_consensus.cc:729] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5986164687a8441f8fbd96cd3e881a34, State: Initialized, Role: FOLLOWER
I20260619 06:37:46.899086 13242 consensus_queue.cc:260] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } }
I20260619 06:37:46.900225 13226 heartbeater.cc:503] Master 127.28.104.254:40281 was elected leader, sending a full tablet report...
I20260619 06:37:46.900857 13242 ts_tablet_manager.cc:1428] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260619 06:37:46.903044 13243 raft_consensus.cc:348] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } }
I20260619 06:37:46.903488 13243 raft_consensus.cc:374] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:37:46.903591 13243 raft_consensus.cc:729] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bb1aa30061484cadc699622de16315, State: Initialized, Role: FOLLOWER
I20260619 06:37:46.904187 13243 consensus_queue.cc:260] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } }
I20260619 06:37:46.906067 13243 ts_tablet_manager.cc:1428] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260619 06:37:46.905833 13244 raft_consensus.cc:348] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } }
I20260619 06:37:46.906267 13244 raft_consensus.cc:374] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:37:46.906353 13244 raft_consensus.cc:729] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f78767d4fa420eb3d659a5c39c68a8, State: Initialized, Role: FOLLOWER
I20260619 06:37:46.906684 13093 heartbeater.cc:503] Master 127.28.104.254:40281 was elected leader, sending a full tablet report...
I20260619 06:37:46.906925 13244 consensus_queue.cc:260] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } }
I20260619 06:37:46.908550 13244 ts_tablet_manager.cc:1428] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260619 06:37:46.980902 13248 raft_consensus.cc:482] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260619 06:37:46.981122 13227 tablet.cc:2369] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:37:46.981321 13248 raft_consensus.cc:504] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } }
I20260619 06:37:46.983280 13248 leader_election.cc:288] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53f78767d4fa420eb3d659a5c39c68a8 (127.28.104.193:45235), 83bb1aa30061484cadc699622de16315 (127.28.104.194:43813)
I20260619 06:37:46.989813 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.983761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.28.104.195:58971)
0619 06:37:46.984142 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:46.984165 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:46.984330 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:46.984744 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:46.984748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:46.984766 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:46.984993 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.985008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.986400 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.986406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.987928 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.987941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.988992 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.988997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.989191 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.989215 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:37:46.989333 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 06:37:46.989481 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":5}
I20260619 06:37:46.989888 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.983698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:58971 (local address 127.28.104.193:45235)
0619 06:37:46.983900 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:46.983908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:46.984286 (+ 378us) server_negotiation.cc:407] Connection header received
0619 06:37:46.984496 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:46.984501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:46.984585 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:46.984690 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:46.985185 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.986277 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.988085 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.988833 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.989002 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.989098 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:46.989487 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 06:37:46.989631 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:37:46.990093 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.983698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:43121 (local address 127.28.104.194:43813)
0619 06:37:46.983912 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:46.983919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:46.984615 (+ 696us) server_negotiation.cc:407] Connection header received
0619 06:37:46.984696 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:46.984701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:46.984770 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:46.984905 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:37:46.985526 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.986515 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.988305 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.989122 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.989314 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.989396 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:46.989694 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 06:37:46.989833 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:37:46.990168 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:46.983795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:43813 (local address 127.28.104.195:43121)
0619 06:37:46.984512 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:46.984524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:37:46.984608 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:37:46.984930 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:37:46.984935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:37:46.984969 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:37:46.985366 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.985377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.986646 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.986652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.988147 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:37:46.988160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:37:46.989331 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:37:46.989338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:37:46.989515 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:46.989547 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:37:46.989634 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:37:46.989750 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":22,"thread_start_us":422,"threads_started":1}
I20260619 06:37:46.991106 13048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d4b1c65514cef98f12eaff463d6c2" candidate_uuid: "5986164687a8441f8fbd96cd3e881a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83bb1aa30061484cadc699622de16315" is_pre_election: true
I20260619 06:37:46.991106 12915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d4b1c65514cef98f12eaff463d6c2" candidate_uuid: "5986164687a8441f8fbd96cd3e881a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f78767d4fa420eb3d659a5c39c68a8" is_pre_election: true
I20260619 06:37:46.991688 12915 raft_consensus.cc:2454] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5986164687a8441f8fbd96cd3e881a34 in term 0.
I20260619 06:37:46.991688 13048 raft_consensus.cc:2454] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5986164687a8441f8fbd96cd3e881a34 in term 0.
I20260619 06:37:46.992396 13115 leader_election.cc:302] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5986164687a8441f8fbd96cd3e881a34, 83bb1aa30061484cadc699622de16315; no voters:
I20260619 06:37:46.992852 13248 raft_consensus.cc:2781] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:37:46.993016 13248 raft_consensus.cc:482] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:37:46.993176 13248 raft_consensus.cc:3037] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:37:46.995288 13248 raft_consensus.cc:504] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } }
I20260619 06:37:46.995889 13248 leader_election.cc:288] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [CANDIDATE]: Term 1 election: Requested vote from peers 53f78767d4fa420eb3d659a5c39c68a8 (127.28.104.193:45235), 83bb1aa30061484cadc699622de16315 (127.28.104.194:43813)
I20260619 06:37:46.996456 13048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d4b1c65514cef98f12eaff463d6c2" candidate_uuid: "5986164687a8441f8fbd96cd3e881a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83bb1aa30061484cadc699622de16315"
I20260619 06:37:46.996510 12915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5d4b1c65514cef98f12eaff463d6c2" candidate_uuid: "5986164687a8441f8fbd96cd3e881a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f78767d4fa420eb3d659a5c39c68a8"
I20260619 06:37:46.996730 12915 raft_consensus.cc:3037] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:37:46.996697 13048 raft_consensus.cc:3037] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:37:46.999151 12915 raft_consensus.cc:2454] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5986164687a8441f8fbd96cd3e881a34 in term 1.
I20260619 06:37:46.999619 13112 leader_election.cc:302] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f78767d4fa420eb3d659a5c39c68a8, 5986164687a8441f8fbd96cd3e881a34; no voters:
I20260619 06:37:46.999687 13048 raft_consensus.cc:2454] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5986164687a8441f8fbd96cd3e881a34 in term 1.
I20260619 06:37:47.000025 13248 raft_consensus.cc:2781] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:37:47.000528 13248 raft_consensus.cc:686] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 1 LEADER]: Becoming Leader. State: Replica: 5986164687a8441f8fbd96cd3e881a34, State: Running, Role: LEADER
I20260619 06:37:47.001009 13248 consensus_queue.cc:237] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } }
I20260619 06:37:47.005923 12773 catalog_manager.cc:5495] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5986164687a8441f8fbd96cd3e881a34 (127.28.104.195). New cstate: current_term: 1 leader_uuid: "5986164687a8441f8fbd96cd3e881a34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986164687a8441f8fbd96cd3e881a34" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 46317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 } health_report { overall_health: UNKNOWN } } }
W20260619 06:37:47.020181 13094 tablet.cc:2369] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:37:47.035382 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.030146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.28.104.195:46317)
0619 06:37:47.030317 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:47.030322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:47.030338 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:37:47.030416 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:47.030419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:47.030500 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:47.030634 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:37:47.031090 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:47.031779 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:47.033448 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:47.033821 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:47.033913 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:47.034413 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:37:47.034886 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:37:47.035014 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:47.035133 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 06:37:47.035193 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:37:47.043793 13048 raft_consensus.cc:1261] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315 [term 1 FOLLOWER]: Refusing update from remote peer 5986164687a8441f8fbd96cd3e881a34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:37:47.043897 12915 raft_consensus.cc:1261] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8 [term 1 FOLLOWER]: Refusing update from remote peer 5986164687a8441f8fbd96cd3e881a34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:37:47.044898 13252 consensus_queue.cc:1035] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83bb1aa30061484cadc699622de16315" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 43813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:37:47.045401 13248 consensus_queue.cc:1035] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f78767d4fa420eb3d659a5c39c68a8" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 45235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:37:47.058611 13257 mvcc.cc:205] Tried to move back new op lower bound from 7298461970601185280 to 7298461970440048640. Current Snapshot: MvccSnapshot[applied={T|T < 7298461970601185280}]
I20260619 06:37:47.076545 12774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:37:47.076884 12774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:37:47.080899 12774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Sending DeleteTablet for 3 replicas of tablet eb5d4b1c65514cef98f12eaff463d6c2
I20260619 06:37:47.082231 12895 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5d4b1c65514cef98f12eaff463d6c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:37:47 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:60160
I20260619 06:37:47.082448 13028 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5d4b1c65514cef98f12eaff463d6c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:37:47 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:50550
I20260619 06:37:47.082473 13159 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5d4b1c65514cef98f12eaff463d6c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:37:47 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:34900
I20260619 06:37:47.082695 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 12811
I20260619 06:37:47.097520 13265 tablet_replica.cc:330] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: stopping tablet replica
I20260619 06:37:47.098097 13265 raft_consensus.cc:2229] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:37:47.098713 13265 raft_consensus.cc:2258] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:37:47.101363 13265 ts_tablet_manager.cc:1905] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:47.108078 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.083744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.28.104.193:45235)
0619 06:37:47.083935 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:47.083941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:47.083962 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:37:47.084097 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:47.084102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:47.084184 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:47.084287 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:47.086552 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:47.087775 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:47.094192 (+ 6417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:47.094932 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:47.095069 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:47.100766 (+ 5697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:47.100783 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:47.100801 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:47.100856 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:47.104899 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:47.105708 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:47.105717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:47.105721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:47.105878 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:47.106290 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:47.106296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:47.106299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:47.106774 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:47.107088 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:47.107401 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 06:37:47.107638 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:37:47.108803 12955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:47.110488 13266 tablet_replica.cc:330] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: stopping tablet replica
I20260619 06:37:47.110846 13267 tablet_replica.cc:330] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: stopping tablet replica
I20260619 06:37:47.111214 13266 raft_consensus.cc:2229] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:37:47.111259 13267 raft_consensus.cc:2229] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:37:47.111769 13267 raft_consensus.cc:2258] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:37:47.112310 13266 raft_consensus.cc:2258] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:37:47.114701 13267 ts_tablet_manager.cc:1905] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:47.114828 13266 ts_tablet_manager.cc:1905] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:37:47.121675 13267 ts_tablet_manager.cc:1918] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:47.121901 13267 log.cc:1196] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/eb5d4b1c65514cef98f12eaff463d6c2
I20260619 06:37:47.122493 13267 ts_tablet_manager.cc:1939] T eb5d4b1c65514cef98f12eaff463d6c2 P 83bb1aa30061484cadc699622de16315: Deleting consensus metadata
I20260619 06:37:47.122880 13266 ts_tablet_manager.cc:1918] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:47.123049 13266 log.cc:1196] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/eb5d4b1c65514cef98f12eaff463d6c2
I20260619 06:37:47.123589 13266 ts_tablet_manager.cc:1939] T eb5d4b1c65514cef98f12eaff463d6c2 P 5986164687a8441f8fbd96cd3e881a34: Deleting consensus metadata
I20260619 06:37:47.124737 12761 catalog_manager.cc:4841] TS 83bb1aa30061484cadc699622de16315 (127.28.104.194:43813): tablet eb5d4b1c65514cef98f12eaff463d6c2 (table test-table [id=6c12bc07412f4014b986a868e9982f86]) successfully deleted
I20260619 06:37:47.125275 12759 catalog_manager.cc:4841] TS 5986164687a8441f8fbd96cd3e881a34 (127.28.104.195:46317): tablet eb5d4b1c65514cef98f12eaff463d6c2 (table test-table [id=6c12bc07412f4014b986a868e9982f86]) successfully deleted
I20260619 06:37:47.238629 13265 ts_tablet_manager.cc:1918] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:37:47.238914 13265 log.cc:1196] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/eb5d4b1c65514cef98f12eaff463d6c2
I20260619 06:37:47.239346 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 12811
I20260619 06:37:47.239502 13265 ts_tablet_manager.cc:1939] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Deleting consensus metadata
W20260619 06:37:47.257599 12757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.104.193:45235 (error 108)
I20260619 06:37:47.257931 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 12964
I20260619 06:37:47.259635 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.259168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.0.0.1:60168)
0619 06:37:47.259339 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:47.259457 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:45235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:37:47.259897 12757 catalog_manager.cc:4584] TS 53f78767d4fa420eb3d659a5c39c68a8 (127.28.104.193:45235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb5d4b1c65514cef98f12eaff463d6c2: Network error: Client connection negotiation failed: client connection to 127.28.104.193:45235: connect: Connection refused (error 111)
I20260619 06:37:47.274619 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.258731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.28.104.194:43813)
0619 06:37:47.258898 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:47.258902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:47.259018 (+ 116us) server_negotiation.cc:407] Connection header received
0619 06:37:47.259205 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:47.259211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:47.259298 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:47.259424 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:47.260974 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:47.261903 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:47.263485 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:47.263866 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:47.263996 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:47.267329 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:47.267342 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:47.267358 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:47.267414 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:47.271446 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:47.272263 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:47.272272 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:47.272276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:47.272425 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:47.272827 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:47.272832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:47.272836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:47.273392 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:47.273684 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:47.274020 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 06:37:47.274257 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:37:47.275476 13088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:47.313828 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.313274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.0.0.1:60182)
0619 06:37:47.313526 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:47.313636 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:45235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 06:37:47.315033 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.314641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.0.0.1:60186)
0619 06:37:47.314797 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:47.314900 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:45235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 06:37:47.365321 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 12964
I20260619 06:37:47.375659 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.375113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.0.0.1:60188)
0619 06:37:47.375357 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:47.375475 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:45235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 06:37:47.376946 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.376484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.0.0.1:60202)
0619 06:37:47.376675 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:47.376788 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:45235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 06:37:47.383345 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 13096
I20260619 06:37:47.397657 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.384095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.28.104.195:46317)
0619 06:37:47.384258 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:37:47.384266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:37:47.384303 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:37:47.384525 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:37:47.384530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:37:47.384593 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:37:47.384706 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:37:47.386126 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:47.386938 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:47.388325 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:37:47.388775 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:37:47.388916 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:37:47.391576 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:37:47.391592 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:37:47.391607 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:37:47.391649 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:37:47.394826 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:47.395473 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:47.395481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:47.395483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:47.395640 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:37:47.396024 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:37:47.396030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:37:47.396033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:37:47.396373 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:37:47.396696 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 06:37:47.396976 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 06:37:47.397231 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:37:47.398393 13221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:47.457803 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.457236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.0.0.1:60206)
0619 06:37:47.457474 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:47.457587 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:45235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 06:37:47.458894 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.458452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.0.0.1:60216)
0619 06:37:47.458654 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:47.458751 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:45235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 06:37:47.520104 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 13096
I20260619 06:37:47.539590 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 12740
I20260619 06:37:47.540766 12804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:37:47.655017 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 12740
I20260619 06:37:47.655094 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:37:47.654365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:45235 (local address 127.0.0.1:60218)
0619 06:37:47.654724 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 06:37:47.654838 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:45235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260619 06:38:07.677075 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:40281
--webserver_interface=127.28.104.254
--webserver_port=44005
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:08.008570 13273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:08.008891 13273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:08.009102 13273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:08.016530 13273 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:38:08.016633 13273 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:38:08.016669 13273 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:38:08.016768 13273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:08.016868 13273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:08.017002 13273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:08.017103 13273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:08.017185 13273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:08.027199 13273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=44005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:08.028517 13273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:08.029953 13273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:08.041992 13278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:08.042083 13281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:08.042102 13279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:08.042562 13273 server_base.cc:1034] running on GCE node
I20260619 06:38:08.043376 13273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:08.044783 13273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:08.046036 13273 hybrid_clock.cc:648] HybridClock initialized: now 1781851088045942 us; error 66 us; skew 500 ppm
I20260619 06:38:08.046439 13273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:08 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851088, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:08.049669 13273 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:38:08.051120 13273 webserver.cc:460] Webserver started at http://127.28.104.254:44005/ using document root <none> and password file <none>
I20260619 06:38:08.051815 13273 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:08.051926 13273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:08.057461 13273 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:38:08.060262 13289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:08.061667 13273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:38:08.061852 13273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "2edcf7f41aa44a1c9ce32011576f2950"
format_stamp: "Formatted at 2026-06-19 06:37:22 on dist-test-slave-nf4n"
server_key: "165d4acafc77c791f94e38496d828f9a"
server_key_iv: "aea1ad70aff81a34e47de2cb7000012d"
server_key_version: "encryptionkey@0"
I20260619 06:38:08.062813 13273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:08.093621 13273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:08.097931 13273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:08.098227 13273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:08.117388 13273 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:40281
I20260619 06:38:08.117431 13340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:40281 every 8 connection(s)
I20260619 06:38:08.118861 13273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:08.120365 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 13273
I20260619 06:38:08.121400 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:45235
--local_ip_for_outbound_sockets=127.28.104.193
--tserver_master_addrs=127.28.104.254:40281
--webserver_port=37811
--webserver_interface=127.28.104.193
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:38:08.129158 13341 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:38:08.133138 13341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Bootstrap starting.
I20260619 06:38:08.154124 13341 log.cc:824] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:08.171869 13341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:38:08.172457 13341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Bootstrap complete.
I20260619 06:38:08.181759 13341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:38:08.182693 13341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edcf7f41aa44a1c9ce32011576f2950, State: Initialized, Role: FOLLOWER
I20260619 06:38:08.183314 13341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:38:08.183563 13341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:08.183683 13341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:08.183876 13341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:38:08.187907 13341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:38:08.188421 13341 leader_election.cc:302] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2edcf7f41aa44a1c9ce32011576f2950; no voters:
I20260619 06:38:08.189182 13341 leader_election.cc:288] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 06:38:08.189321 13345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:38:08.190480 13345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [term 3 LEADER]: Becoming Leader. State: Replica: 2edcf7f41aa44a1c9ce32011576f2950, State: Running, Role: LEADER
I20260619 06:38:08.191102 13345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } }
I20260619 06:38:08.191449 13341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:08.194095 13347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2edcf7f41aa44a1c9ce32011576f2950. Latest consensus state: current_term: 3 leader_uuid: "2edcf7f41aa44a1c9ce32011576f2950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } } }
I20260619 06:38:08.194551 13347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:08.195092 13346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2edcf7f41aa44a1c9ce32011576f2950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcf7f41aa44a1c9ce32011576f2950" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 40281 } } }
I20260619 06:38:08.195411 13346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:08.206671 13351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:08.216569 13351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:08.218322 13351 catalog_manager.cc:1259] Loaded cluster ID: 33217898ca564c43969c38dce61ad3e8
I20260619 06:38:08.218511 13351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:08.222972 13351 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:08.224748 13351 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Loaded TSK: 0
I20260619 06:38:08.225057 13351 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2edcf7f41aa44a1c9ce32011576f2950: Loaded TSK: 1
F20260619 06:38:08.226368 13351 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f56c82299ad google::LogMessage::Fail() at ??:0
@ 0x7f56c822c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f56c82294de google::LogMessage::Flush() at ??:0
@ 0x7f56c822aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f56cbf8d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f56cbf7c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f56ddc581c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f56ddc57783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f56ddce32a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f56ddce30c4 std::_Function_handler<>::_M_invoke() at ??:0
I20260619 06:38:08.266784 13363 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f56ddd2d5e9 std::function<>::operator()() at ??:0
@ 0x7f56ddc5bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f56ddc59c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f56ddce1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f56ddce1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f56cae75345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f56cae7891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f56cae78799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f56cae53eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f56ce12d6db start_thread at ??:0
@ 0x7f56c70c361f clone at ??:0
W20260619 06:38:08.544194 13343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:08.544463 13343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:08.544557 13343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:08.552004 13343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:08.552114 13343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:08.552227 13343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:38:08.562173 13343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:45235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=37811
--tserver_master_addrs=127.28.104.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:08.563426 13343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:08.564667 13343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:08.577778 13389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:08.577936 13392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:08.577828 13390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:08.578881 13343 server_base.cc:1034] running on GCE node
I20260619 06:38:08.579460 13343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:08.580436 13343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:08.581671 13343 hybrid_clock.cc:648] HybridClock initialized: now 1781851088581611 us; error 46 us; skew 500 ppm
I20260619 06:38:08.581940 13343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:08 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851088, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:08.584911 13343 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:38:08.586395 13343 webserver.cc:460] Webserver started at http://127.28.104.193:37811/ using document root <none> and password file <none>
I20260619 06:38:08.587069 13343 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:08.587218 13343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:08.592589 13343 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:38:08.595254 13399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:08.596483 13343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:38:08.596658 13343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "53f78767d4fa420eb3d659a5c39c68a8"
format_stamp: "Formatted at 2026-06-19 06:37:22 on dist-test-slave-nf4n"
server_key: "f6ad9a33faf7bfec1e71ba78422e49b7"
server_key_iv: "49d4fc2605b731f23338ce4a47eb462a"
server_key_version: "encryptionkey@0"
I20260619 06:38:08.597677 13343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:08.613350 13343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:08.617198 13343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:08.617483 13343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:08.618468 13343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:08.620199 13406 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 06:38:08.625331 13406 ts_tablet_manager.cc:1852] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 06:38:08.626297 13406 ts_tablet_manager.cc:1905] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:08.633153 13406 ts_tablet_manager.cc:1918] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:08.633447 13406 ts_tablet_manager.cc:1939] T eb5d4b1c65514cef98f12eaff463d6c2 P 53f78767d4fa420eb3d659a5c39c68a8: Deleting consensus metadata
I20260619 06:38:08.633855 13343 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 06:38:08.633992 13343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260619 06:38:08.634126 13343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:08.634214 13343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:08.667718 13343 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:45235
I20260619 06:38:08.667809 13512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:45235 every 8 connection(s)
I20260619 06:38:08.669123 13343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:38:08.669574 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 13343
I20260619 06:38:08.670485 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:43813
--local_ip_for_outbound_sockets=127.28.104.194
--tserver_master_addrs=127.28.104.254:40281
--webserver_port=34743
--webserver_interface=127.28.104.194
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:38:08.674979 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:08.672723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:48893)
0619 06:38:08.673687 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:08.673971 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":412,"threads_started":1}
W20260619 06:38:08.675916 13513 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:40281 (0 consecutive failures): Network error: Failed to ping master at 127.28.104.254:40281: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
I20260619 06:38:08.676821 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:08.676394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:42157)
0619 06:38:08.676610 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:08.676686 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 06:38:08.678176 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:08.677326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:55377)
0619 06:38:08.677892 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:08.677977 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":272,"threads_started":1}
W20260619 06:38:08.678391 13513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:38:09.012281 13518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:09.012651 13518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:09.012830 13518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:09.020272 13518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:09.020437 13518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:09.020632 13518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:38:09.030548 13518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:43813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=34743
--tserver_master_addrs=127.28.104.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:09.032115 13518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:09.033550 13518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:09.046710 13523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:09.046877 13526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:09.047396 13524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:09.048563 13518 server_base.cc:1034] running on GCE node
I20260619 06:38:09.049221 13518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:09.050105 13518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:09.051330 13518 hybrid_clock.cc:648] HybridClock initialized: now 1781851089051296 us; error 29 us; skew 500 ppm
I20260619 06:38:09.051632 13518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:09 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851089, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:09.054773 13518 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:38:09.056304 13518 webserver.cc:460] Webserver started at http://127.28.104.194:34743/ using document root <none> and password file <none>
I20260619 06:38:09.057073 13518 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:09.057222 13518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:09.062453 13518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:38:09.065186 13533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:09.066624 13518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:38:09.066793 13518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "83bb1aa30061484cadc699622de16315"
format_stamp: "Formatted at 2026-06-19 06:37:23 on dist-test-slave-nf4n"
server_key: "33e59c0cccd333f02db09948c292e9e0"
server_key_iv: "ca46bde28831ee74eae75c39fa51c9aa"
server_key_version: "encryptionkey@0"
I20260619 06:38:09.067741 13518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:09.083923 13518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:09.088030 13518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:09.088335 13518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:09.089426 13518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:09.090678 13518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:09.090797 13518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:09.090929 13518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:09.091012 13518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:09.124320 13518 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:43813
I20260619 06:38:09.124414 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:43813 every 8 connection(s)
I20260619 06:38:09.125854 13518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:38:09.130750 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:09.128922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:52365)
0619 06:38:09.129676 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:09.129928 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":267,"threads_started":1}
W20260619 06:38:09.131323 13646 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:40281 (0 consecutive failures): Network error: Failed to ping master at 127.28.104.254:40281: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
I20260619 06:38:09.132040 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:09.131762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:36871)
0619 06:38:09.131885 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:09.131945 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:38:09.132819 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:09.132510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:54341)
0619 06:38:09.132642 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:09.132719 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:38:09.133093 13646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:38:09.135712 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 13518
I20260619 06:38:09.136297 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:46317
--local_ip_for_outbound_sockets=127.28.104.195
--tserver_master_addrs=127.28.104.254:40281
--webserver_port=43683
--webserver_interface=127.28.104.195
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:09.460062 13649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:09.460500 13649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:09.460691 13649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:09.468132 13649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:09.468303 13649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:09.468482 13649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:38:09.478564 13649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:46317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=43683
--tserver_master_addrs=127.28.104.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:09.479905 13649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:09.481257 13649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:09.494208 13654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:09.494172 13655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:09.494243 13657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:09.495003 13649 server_base.cc:1034] running on GCE node
I20260619 06:38:09.495589 13649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:09.496491 13649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:09.497707 13649 hybrid_clock.cc:648] HybridClock initialized: now 1781851089497631 us; error 70 us; skew 500 ppm
I20260619 06:38:09.498032 13649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:09 dist-test-slave-nf4n krb5kdc[12186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851089, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:09.501173 13649 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:38:09.502621 13649 webserver.cc:460] Webserver started at http://127.28.104.195:43683/ using document root <none> and password file <none>
I20260619 06:38:09.503257 13649 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:09.503382 13649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:09.508514 13649 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:38:09.511116 13664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:09.512406 13649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:38:09.512579 13649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "5986164687a8441f8fbd96cd3e881a34"
format_stamp: "Formatted at 2026-06-19 06:37:23 on dist-test-slave-nf4n"
server_key: "27d82cafc8bf57f8c8bca5ce47faea72"
server_key_iv: "26ef92f102357a9b414111161e1e45b1"
server_key_version: "encryptionkey@0"
I20260619 06:38:09.513586 13649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:09.528014 13649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:09.531953 13649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:09.532215 13649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:09.533211 13649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:09.534463 13649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:09.534579 13649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:09.534699 13649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:09.534792 13649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:09.567737 13649 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:46317
I20260619 06:38:09.567832 13777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:46317 every 8 connection(s)
I20260619 06:38:09.569514 13649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:38:09.574638 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:09.572708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:43963)
0619 06:38:09.573560 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:09.573827 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":314,"threads_started":1}
W20260619 06:38:09.575213 13778 heartbeater.cc:644] Failed to heartbeat to 127.28.104.254:40281 (0 consecutive failures): Network error: Failed to ping master at 127.28.104.254:40281: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
I20260619 06:38:09.575941 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:09.575642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:45199)
0619 06:38:09.575780 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:09.575844 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:38:09.576649 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:09.576376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:36403)
0619 06:38:09.576489 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:09.576554 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 06:38:09.576874 13778 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:38:09.578402 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 13649
I20260619 06:38:09.679916 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:09.679040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:46841)
0619 06:38:09.679527 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:09.679610 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":227,"threads_started":1}
I20260619 06:38:10.134336 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:10.133576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:54609)
0619 06:38:10.134015 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:10.134119 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":203,"threads_started":1}
I20260619 06:38:10.578262 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:10.577490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:45595)
0619 06:38:10.577937 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:10.578043 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260619 06:38:10.681550 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:10.680812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:60307)
0619 06:38:10.681293 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:10.681381 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":285,"threads_started":1}
I20260619 06:38:11.135875 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:11.135150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:57033)
0619 06:38:11.135636 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:11.135717 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
I20260619 06:38:11.579937 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:11.579117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:44217)
0619 06:38:11.579620 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:11.579743 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":251,"threads_started":1}
I20260619 06:38:11.683072 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:11.682415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:33501)
0619 06:38:11.682833 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:11.682914 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":217,"threads_started":1}
I20260619 06:38:12.137480 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:12.136737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:47603)
0619 06:38:12.137205 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:12.137313 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":226,"threads_started":1}
I20260619 06:38:12.581490 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:12.580778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:55559)
0619 06:38:12.581250 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:12.581326 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":261,"threads_started":1}
I20260619 06:38:12.684644 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:12.683969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:60431)
0619 06:38:12.684390 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:12.684470 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":215,"threads_started":1}
I20260619 06:38:13.138991 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:13.138309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:50527)
0619 06:38:13.138724 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:13.138803 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":202,"threads_started":1}
I20260619 06:38:13.583050 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:13.582333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:41533)
0619 06:38:13.582787 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:13.582868 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":223,"threads_started":1}
I20260619 06:38:13.686170 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:13.685518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:35911)
0619 06:38:13.685943 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:13.686020 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":207,"threads_started":1}
I20260619 06:38:14.140524 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:14.139858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:34695)
0619 06:38:14.140291 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:14.140371 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":230,"threads_started":1}
I20260619 06:38:14.584692 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:14.583927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:55765)
0619 06:38:14.584449 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:14.584528 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260619 06:38:14.687768 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:14.687063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:59825)
0619 06:38:14.687496 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:14.687589 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260619 06:38:15.142035 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:15.141386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:52393)
0619 06:38:15.141789 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:15.141880 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":204,"threads_started":1}
I20260619 06:38:15.586364 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:15.585662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:35881)
0619 06:38:15.586107 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:15.586196 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":252,"threads_started":1}
I20260619 06:38:15.689482 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:15.688627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:45837)
0619 06:38:15.689199 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:15.689285 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":243,"threads_started":1}
I20260619 06:38:16.143637 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:16.142882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:50487)
0619 06:38:16.143355 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:16.143433 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":211,"threads_started":1}
I20260619 06:38:16.588073 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:16.587325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:51499)
0619 06:38:16.587783 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:16.587867 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":207,"threads_started":1}
I20260619 06:38:16.691087 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:16.690271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:56779)
0619 06:38:16.690790 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:16.690885 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":243,"threads_started":1}
I20260619 06:38:17.145218 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:17.144539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:59941)
0619 06:38:17.144957 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:17.145055 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":212,"threads_started":1}
I20260619 06:38:17.590071 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:17.589220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:34861)
0619 06:38:17.589716 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:17.589851 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":273,"threads_started":1}
I20260619 06:38:17.692606 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:17.691898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:36349)
0619 06:38:17.692359 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:17.692436 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":226,"threads_started":1}
I20260619 06:38:18.146833 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:18.146135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:34031)
0619 06:38:18.146585 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:18.146667 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":211,"threads_started":1}
I20260619 06:38:18.591692 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:18.590982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:59037)
0619 06:38:18.591443 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:18.591524 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":214,"threads_started":1}
I20260619 06:38:18.694281 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:18.693570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:48393)
0619 06:38:18.694027 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:18.694106 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":260,"threads_started":1}
I20260619 06:38:19.148427 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:19.147762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:35889)
0619 06:38:19.148191 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:19.148269 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":233,"threads_started":1}
I20260619 06:38:19.593396 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:19.592677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:51587)
0619 06:38:19.593150 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:19.593231 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260619 06:38:19.695870 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:19.695184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:57287)
0619 06:38:19.695598 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:19.695684 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":219,"threads_started":1}
I20260619 06:38:20.150056 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:20.149346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:34097)
0619 06:38:20.149795 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:20.149884 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
I20260619 06:38:20.594941 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:20.594254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:58207)
0619 06:38:20.594667 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:20.594756 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":212,"threads_started":1}
I20260619 06:38:20.697458 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:20.696685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:34755)
0619 06:38:20.697180 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:20.697262 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":239,"threads_started":1}
I20260619 06:38:21.151566 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:21.150918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:45193)
0619 06:38:21.151340 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:21.151419 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":218,"threads_started":1}
I20260619 06:38:21.596618 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:21.595949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:56203)
0619 06:38:21.596383 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:21.596463 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":227,"threads_started":1}
I20260619 06:38:21.699057 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:21.698349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:38781)
0619 06:38:21.698790 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:21.698888 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":218,"threads_started":1}
I20260619 06:38:22.153326 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:22.152533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:56945)
0619 06:38:22.153046 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:22.153143 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":256,"threads_started":1}
I20260619 06:38:22.598378 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:22.597539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:46025)
0619 06:38:22.598093 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:22.598192 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":301,"threads_started":1}
I20260619 06:38:22.700699 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:22.699948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:54693)
0619 06:38:22.700419 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:22.700502 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":228,"threads_started":1}
I20260619 06:38:23.154964 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:23.154273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:56767)
0619 06:38:23.154696 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:23.154780 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":222,"threads_started":1}
I20260619 06:38:23.599980 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:23.599279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:43621)
0619 06:38:23.599692 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:23.599776 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
I20260619 06:38:23.702414 13825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:23.701717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:52341)
0619 06:38:23.702148 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:23.702239 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":212,"threads_started":1}
I20260619 06:38:24.156687 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:24.156005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.194:48349)
0619 06:38:24.156418 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:24.156510 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
I20260619 06:38:24.590258 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 13343
I20260619 06:38:24.602172 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:24.601179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.195:34591)
0619 06:38:24.601727 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:24.601858 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":257,"threads_started":1}
I20260619 06:38:24.619521 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:24.591341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.28.104.193:45235)
0619 06:38:24.592103 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:24.592125 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:24.592176 (+ 51us) server_negotiation.cc:407] Connection header received
0619 06:38:24.594080 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:24.594102 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:24.594522 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:24.594918 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:24.596365 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:24.597671 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:24.598943 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:24.599605 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:24.599840 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:24.608917 (+ 9077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:24.608931 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:24.608960 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:24.609043 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:24.613707 (+ 4664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:24.614827 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:24.614836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:24.614846 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:24.615081 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:24.615617 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:24.615624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:24.615630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:24.616563 (+ 933us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:24.618112 (+ 1549us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:24.618455 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 06:38:24.618980 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":380,"threads_started":1}
I20260619 06:38:24.621651 13508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:24.723578 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:24.722081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:40281 (local address 127.28.104.193:37851)
0619 06:38:24.723046 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:24.723279 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.254:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":635,"threads_started":1}
I20260619 06:38:24.725378 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 13343
I20260619 06:38:24.742339 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 13518
I20260619 06:38:24.759402 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:24.743242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.28.104.194:43813)
0619 06:38:24.743814 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:24.743828 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:24.743870 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:38:24.745535 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:24.745546 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:24.745985 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:24.746290 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:24.747578 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:24.748847 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:24.750101 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:24.750895 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:24.751046 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:24.753592 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:24.753599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:24.753614 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:24.753666 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:24.756553 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:24.757162 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:24.757168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:24.757173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:24.757355 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:24.757688 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:24.757691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:24.757693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:24.758172 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:24.758428 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:24.758792 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 06:38:24.759103 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":249,"threads_started":1}
I20260619 06:38:24.760725 13641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:24.847822 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 13518
I20260619 06:38:24.865340 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 13649
I20260619 06:38:24.883759 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:24.866373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.28.104.195:46317)
0619 06:38:24.867009 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:24.867024 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:24.867066 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:38:24.868681 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:24.868690 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:24.869203 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:24.869491 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:24.870769 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:24.872027 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:24.873460 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:24.874324 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:24.874477 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:24.877069 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:24.877088 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:24.877106 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:24.877152 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:24.880107 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:24.880874 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:24.880881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:24.880888 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:24.881131 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:24.881599 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:24.881605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:24.881608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:24.882333 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:24.882582 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:24.882993 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 06:38:24.883461 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":297,"threads_started":1}
I20260619 06:38:24.885232 13773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:24.968621 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 13649
2026-06-19T06:38:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64436 ms)
[----------] 30 tests from SecurityITest (299412 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:38:25 dist-test-slave-nf4n krb5kdc[13837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:38:25 dist-test-slave-nf4n krb5kdc[13837](info): set up 2 sockets
Jun 19 06:38:25 dist-test-slave-nf4n krb5kdc[13837](info): commencing operation
krb5kdc: starting...
W20260619 06:38:26.081259 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:38:26 dist-test-slave-nf4n krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:38:26.322216 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:32971
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:32971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:26.646818 13850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:26.647152 13850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:26.647303 13850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:26.654479 13850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:26.654615 13850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:26.654732 13850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:26.654832 13850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:26.654906 13850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:26.664462 13850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:32971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:26.665776 13850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:26.667059 13850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:26.678593 13858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:26.678668 13856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:26.678690 13855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:26.679365 13850 server_base.cc:1034] running on GCE node
I20260619 06:38:26.680040 13850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:26.681365 13850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:26.682592 13850 hybrid_clock.cc:648] HybridClock initialized: now 1781851106682549 us; error 48 us; skew 500 ppm
I20260619 06:38:26.682934 13850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:26 dist-test-slave-nf4n krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:26.685885 13850 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:38:26.687294 13850 webserver.cc:460] Webserver started at http://127.28.104.254:45025/ using document root <none> and password file <none>
I20260619 06:38:26.687968 13850 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:26.688109 13850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:26.688416 13850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:26.691331 13850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "862c1845d28f4686b900dc5fc69ffcb2"
format_stamp: "Formatted at 2026-06-19 06:38:26 on dist-test-slave-nf4n"
server_key: "cde0429e5821f78d15bdbf404b2a128d"
server_key_iv: "380fb2677057eb677b5cad11ffff4cc9"
server_key_version: "encryptionkey@0"
I20260619 06:38:26.692055 13850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "862c1845d28f4686b900dc5fc69ffcb2"
format_stamp: "Formatted at 2026-06-19 06:38:26 on dist-test-slave-nf4n"
server_key: "cde0429e5821f78d15bdbf404b2a128d"
server_key_iv: "380fb2677057eb677b5cad11ffff4cc9"
server_key_version: "encryptionkey@0"
I20260619 06:38:26.696884 13850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260619 06:38:26.700109 13865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:26.701471 13850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:38:26.701643 13850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "862c1845d28f4686b900dc5fc69ffcb2"
format_stamp: "Formatted at 2026-06-19 06:38:26 on dist-test-slave-nf4n"
server_key: "cde0429e5821f78d15bdbf404b2a128d"
server_key_iv: "380fb2677057eb677b5cad11ffff4cc9"
server_key_version: "encryptionkey@0"
I20260619 06:38:26.701829 13850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:26.716038 13850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:26.719946 13850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:26.720229 13850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:26.737749 13850 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:32971
I20260619 06:38:26.737767 13916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:32971 every 8 connection(s)
I20260619 06:38:26.739110 13850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:26.742641 13917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:26.744902 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 13850
I20260619 06:38:26.745153 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:26.746064 29091 external_mini_cluster.cc:1428] Setting key e7ca68b4720bdda73f97956a610038a7
I20260619 06:38:26.754992 13917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2: Bootstrap starting.
I20260619 06:38:26.758701 13917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:26.759966 13917 log.cc:824] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:38:26 dist-test-slave-nf4n krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:26.763265 13917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2: No bootstrap required, opened a new log
I20260619 06:38:26.771560 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:26.748227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.28.104.254:32971)
0619 06:38:26.749160 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:26.749173 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:26.749219 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:38:26.750970 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:26.750991 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:26.751400 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:26.751747 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:26.753532 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:26.754912 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:26.756861 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:26.757615 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:26.757812 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:26.762488 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:26.762501 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:26.762521 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:26.762574 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:26.766857 (+ 4283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:26.767519 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:26.767528 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:26.767537 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:26.767775 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:26.768149 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:26.768154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:26.768159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:26.768929 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:26.769306 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:26.769655 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 06:38:26.770115 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":429,"threads_started":1}
I20260619 06:38:26.775207 13917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c1845d28f4686b900dc5fc69ffcb2" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32971 } }
I20260619 06:38:26.775761 13917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:26.775902 13917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862c1845d28f4686b900dc5fc69ffcb2, State: Initialized, Role: FOLLOWER
I20260619 06:38:26.776687 13917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c1845d28f4686b900dc5fc69ffcb2" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32971 } }
I20260619 06:38:26.777069 13917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:26.777278 13917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:26.777477 13917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:26.780980 13917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c1845d28f4686b900dc5fc69ffcb2" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32971 } }
I20260619 06:38:26.781553 13917 leader_election.cc:302] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 862c1845d28f4686b900dc5fc69ffcb2; no voters:
I20260619 06:38:26.782320 13917 leader_election.cc:288] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:26.782429 13922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:26.783104 13922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [term 1 LEADER]: Becoming Leader. State: Replica: 862c1845d28f4686b900dc5fc69ffcb2, State: Running, Role: LEADER
I20260619 06:38:26.783661 13922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c1845d28f4686b900dc5fc69ffcb2" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32971 } }
I20260619 06:38:26.784561 13917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:26.786737 13924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 862c1845d28f4686b900dc5fc69ffcb2. Latest consensus state: current_term: 1 leader_uuid: "862c1845d28f4686b900dc5fc69ffcb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c1845d28f4686b900dc5fc69ffcb2" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32971 } } }
I20260619 06:38:26.786708 13923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "862c1845d28f4686b900dc5fc69ffcb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c1845d28f4686b900dc5fc69ffcb2" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32971 } } }
I20260619 06:38:26.787182 13924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:26.787225 13923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:26.788700 13928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:26.794440 13928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:26.803539 13928 catalog_manager.cc:1347] Generated new cluster ID: 57b01ed5d9924cd28da95904a91ad901
I20260619 06:38:26.803658 13928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:26.825889 13928 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:26.827199 13928 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:26.834084 13928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2: Generated new TSK 0
I20260619 06:38:26.834894 13928 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:38:26.896042 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:32971
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:27.248560 13943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:27.248847 13943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:27.248946 13943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:27.256253 13943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:27.256358 13943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:27.256471 13943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:38:27.266306 13943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:32971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:27.267482 13943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:27.268862 13943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:27.281600 13949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:27.281560 13951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:27.281627 13948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:27.282780 13943 server_base.cc:1034] running on GCE node
I20260619 06:38:27.283380 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:27.284387 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:27.285604 13943 hybrid_clock.cc:648] HybridClock initialized: now 1781851107285550 us; error 67 us; skew 500 ppm
I20260619 06:38:27.285969 13943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:27 dist-test-slave-nf4n krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:27.289127 13943 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:38:27.290510 13943 webserver.cc:460] Webserver started at http://127.28.104.193:39603/ using document root <none> and password file <none>
I20260619 06:38:27.291185 13943 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:27.291313 13943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:27.291623 13943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:27.294513 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "886ceb12a99c4c73b286f9eb602e8361"
format_stamp: "Formatted at 2026-06-19 06:38:27 on dist-test-slave-nf4n"
server_key: "915f2de7abb5e729af376c2ec5e76bca"
server_key_iv: "8607983f08dd42a64ebd4c82918b92bf"
server_key_version: "encryptionkey@0"
I20260619 06:38:27.295260 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "886ceb12a99c4c73b286f9eb602e8361"
format_stamp: "Formatted at 2026-06-19 06:38:27 on dist-test-slave-nf4n"
server_key: "915f2de7abb5e729af376c2ec5e76bca"
server_key_iv: "8607983f08dd42a64ebd4c82918b92bf"
server_key_version: "encryptionkey@0"
I20260619 06:38:27.300372 13943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 06:38:27.303686 13958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:27.305087 13943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:38:27.305243 13943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "886ceb12a99c4c73b286f9eb602e8361"
format_stamp: "Formatted at 2026-06-19 06:38:27 on dist-test-slave-nf4n"
server_key: "915f2de7abb5e729af376c2ec5e76bca"
server_key_iv: "8607983f08dd42a64ebd4c82918b92bf"
server_key_version: "encryptionkey@0"
I20260619 06:38:27.305440 13943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:27.330842 13943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:27.335443 13943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:27.335750 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:27.336716 13943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:27.338101 13943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:27.338174 13943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:27.338339 13943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:27.338395 13943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:27.371806 13943 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:36605
I20260619 06:38:27.371856 14070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:36605 every 8 connection(s)
I20260619 06:38:27.373181 13943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:38:27.379673 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 13943
I20260619 06:38:27.379894 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:38:27.380514 29091 external_mini_cluster.cc:1428] Setting key bb7507cd819fcd03851d4604efcd41e0
Jun 19 06:38:27 dist-test-slave-nf4n krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:27.394508 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:27.375662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:40643 (local address 127.28.104.254:32971)
0619 06:38:27.376124 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:27.376131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:27.377092 (+ 961us) server_negotiation.cc:407] Connection header received
0619 06:38:27.378914 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:27.378918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:27.378985 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:27.379080 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:27.380704 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:27.381584 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:27.383811 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:27.384249 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:27.384354 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:27.388540 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:27.388546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:27.388550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:27.388585 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:27.391389 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:27.392129 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:27.392135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:27.392139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:27.392260 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:27.392723 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:27.392730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:27.392734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:27.393001 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:27.393207 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:27.393916 (+ 709us) server_negotiation.cc:299] Negotiation successful
0619 06:38:27.394143 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":218,"threads_started":1}
I20260619 06:38:27.395553 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:27.376236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:32971 (local address 127.28.104.193:40643)
0619 06:38:27.376929 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:27.376965 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:27.378661 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:27.379281 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:27.379289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:27.379899 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:27.380485 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:27.380502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:27.381820 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:27.381825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:27.383655 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:27.383678 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:27.384485 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:27.384494 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:27.384830 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:27.385624 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:38:27.385659 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:27.388312 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:27.391535 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:27.391545 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:27.391564 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:27.392004 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:27.392377 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:27.392383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:27.392387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:27.392608 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:27.393178 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:27.393186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:27.393535 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 06:38:27.393818 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 06:38:27.394361 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":260,"threads_started":1}
I20260619 06:38:27.397106 14071 heartbeater.cc:347] Connected to a master server at 127.28.104.254:32971
I20260619 06:38:27.397434 14071 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:27.398308 14071 heartbeater.cc:511] Master 127.28.104.254:32971 requested a full tablet report, sending...
I20260619 06:38:27.400619 13882 ts_manager.cc:194] Registered new tserver with Master: 886ceb12a99c4c73b286f9eb602e8361 (127.28.104.193:36605)
I20260619 06:38:27.402133 13882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:40643
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:38:27.436512 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:32971
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:27.752996 13913 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.28.104.254:32971
W20260619 06:38:27.788630 14077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:27.788920 14077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:27.789039 14077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:27.797219 14077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:27.797330 14077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:27.797446 14077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:38:27.807583 14077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:32971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:27.808904 14077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:27.810595 14077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:27.823613 14082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:27.823670 14085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:27.823578 14083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:27.824745 14077 server_base.cc:1034] running on GCE node
I20260619 06:38:27.825394 14077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:27.826313 14077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:27.827538 14077 hybrid_clock.cc:648] HybridClock initialized: now 1781851107827493 us; error 60 us; skew 500 ppm
I20260619 06:38:27.827858 14077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:27 dist-test-slave-nf4n krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:27.831063 14077 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:38:27.832445 14077 webserver.cc:460] Webserver started at http://127.28.104.194:44949/ using document root <none> and password file <none>
I20260619 06:38:27.833213 14077 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:27.833325 14077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:27.833582 14077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:27.836479 14077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "46a24827cc764d25bd377c1934bde6d7"
format_stamp: "Formatted at 2026-06-19 06:38:27 on dist-test-slave-nf4n"
server_key: "515c9127cee09f470a68c26257d3b385"
server_key_iv: "b171c5ba23a8d4aa54f575d3454ae0ed"
server_key_version: "encryptionkey@0"
I20260619 06:38:27.837252 14077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "46a24827cc764d25bd377c1934bde6d7"
format_stamp: "Formatted at 2026-06-19 06:38:27 on dist-test-slave-nf4n"
server_key: "515c9127cee09f470a68c26257d3b385"
server_key_iv: "b171c5ba23a8d4aa54f575d3454ae0ed"
server_key_version: "encryptionkey@0"
I20260619 06:38:27.842250 14077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:38:27.845578 14092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:27.846882 14077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:38:27.847052 14077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "46a24827cc764d25bd377c1934bde6d7"
format_stamp: "Formatted at 2026-06-19 06:38:27 on dist-test-slave-nf4n"
server_key: "515c9127cee09f470a68c26257d3b385"
server_key_iv: "b171c5ba23a8d4aa54f575d3454ae0ed"
server_key_version: "encryptionkey@0"
I20260619 06:38:27.847254 14077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:27.879601 14077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:27.883790 14077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:27.884075 14077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:27.885227 14077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:27.886649 14077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:27.886730 14077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:27.886905 14077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:27.886955 14077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:27.923743 14077 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:35591
I20260619 06:38:27.923833 14206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:35591 every 8 connection(s)
I20260619 06:38:27.925246 14077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:38:27.932879 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 14077
I20260619 06:38:27.933084 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:38:27.933657 29091 external_mini_cluster.cc:1428] Setting key 7b76bb0de4cab56d2042e8487df999af
Jun 19 06:38:27 dist-test-slave-nf4n krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:27.947105 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:27.927868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:37211 (local address 127.28.104.254:32971)
0619 06:38:27.928306 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:27.928311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:27.929345 (+ 1034us) server_negotiation.cc:407] Connection header received
0619 06:38:27.931042 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:27.931046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:27.931128 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:27.931226 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:27.932742 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:27.933912 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:27.935716 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:27.936381 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:27.936544 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:27.939971 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:27.939980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:27.939988 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:27.940035 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:27.943673 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:27.944716 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:27.944720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:27.944723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:27.944853 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:27.945396 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:27.945399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:27.945401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:27.945663 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:27.945772 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:27.946661 (+ 889us) server_negotiation.cc:299] Negotiation successful
0619 06:38:27.946836 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":213,"threads_started":1}
I20260619 06:38:27.948659 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:27.928437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:32971 (local address 127.28.104.194:37211)
0619 06:38:27.929171 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:27.929236 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:27.930809 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:27.931474 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:27.931482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:27.931989 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:27.932572 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:27.932589 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:27.934059 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:27.934063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:27.935587 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:27.935599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:27.936594 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:27.936599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:27.936794 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:27.937525 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:38:27.937553 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:27.939704 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:27.943822 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:27.943829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:27.943842 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:27.944582 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:27.944963 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:27.944971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:27.944977 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:27.945290 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:27.945790 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:27.945801 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:27.946283 (+ 482us) client_negotiation.cc:769] Sending connection context
0619 06:38:27.946586 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 06:38:27.947160 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":277,"threads_started":1}
I20260619 06:38:27.950194 14207 heartbeater.cc:347] Connected to a master server at 127.28.104.254:32971
I20260619 06:38:27.950529 14207 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:27.951320 14207 heartbeater.cc:511] Master 127.28.104.254:32971 requested a full tablet report, sending...
I20260619 06:38:27.952793 13882 ts_manager.cc:194] Registered new tserver with Master: 46a24827cc764d25bd377c1934bde6d7 (127.28.104.194:35591)
I20260619 06:38:27.953663 13882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:37211
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:38:27.988487 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:32971
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:28.313862 14213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:28.314141 14213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:28.314294 14213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:28.321818 14213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:28.321921 14213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:28.322095 14213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:38:28.331887 14213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:32971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:28.333199 14213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:28.334547 14213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:28.347344 14218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:28.347411 14221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:28.347386 14219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:28.348100 14213 server_base.cc:1034] running on GCE node
I20260619 06:38:28.348685 14213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:28.349687 14213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:28.350942 14213 hybrid_clock.cc:648] HybridClock initialized: now 1781851108350889 us; error 30 us; skew 500 ppm
I20260619 06:38:28.351307 14213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:28 dist-test-slave-nf4n krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:28.354808 14213 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:38:28.356263 14213 webserver.cc:460] Webserver started at http://127.28.104.195:40833/ using document root <none> and password file <none>
I20260619 06:38:28.356957 14213 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:28.357123 14213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:28.357435 14213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:28.360282 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "0ea4b217fe3c4c299d6477766e68f68d"
format_stamp: "Formatted at 2026-06-19 06:38:28 on dist-test-slave-nf4n"
server_key: "3c03397190bd6013f50833be9032c7a3"
server_key_iv: "75e8b7e7e0bdde021cab396ee52859ea"
server_key_version: "encryptionkey@0"
I20260619 06:38:28.361068 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "0ea4b217fe3c4c299d6477766e68f68d"
format_stamp: "Formatted at 2026-06-19 06:38:28 on dist-test-slave-nf4n"
server_key: "3c03397190bd6013f50833be9032c7a3"
server_key_iv: "75e8b7e7e0bdde021cab396ee52859ea"
server_key_version: "encryptionkey@0"
I20260619 06:38:28.366010 14213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 06:38:28.369295 14229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:28.370463 14213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:38:28.370630 14213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "0ea4b217fe3c4c299d6477766e68f68d"
format_stamp: "Formatted at 2026-06-19 06:38:28 on dist-test-slave-nf4n"
server_key: "3c03397190bd6013f50833be9032c7a3"
server_key_iv: "75e8b7e7e0bdde021cab396ee52859ea"
server_key_version: "encryptionkey@0"
I20260619 06:38:28.370850 14213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:28.397007 14213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:28.401505 14213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:28.401787 14213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:28.402707 14213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:28.404039 14213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:28.404139 14213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:28.404301 14213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:28.404386 14213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:28.405830 14071 heartbeater.cc:503] Master 127.28.104.254:32971 was elected leader, sending a full tablet report...
I20260619 06:38:28.441288 14213 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:34371
I20260619 06:38:28.441329 14342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:34371 every 8 connection(s)
I20260619 06:38:28.442605 14213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:38:28.444471 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 14213
I20260619 06:38:28.444686 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:38:28.445313 29091 external_mini_cluster.cc:1428] Setting key 1629135bba974a39df221994ba18ed89
Jun 19 06:38:28 dist-test-slave-nf4n krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851108, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:28.462375 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.445269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:47433 (local address 127.28.104.254:32971)
0619 06:38:28.445468 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:28.445481 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:28.446926 (+ 1445us) server_negotiation.cc:407] Connection header received
0619 06:38:28.448942 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:28.448948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:28.449076 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:28.449185 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:28.450705 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.451462 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.453006 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.453471 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.453605 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.456897 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:28.456902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:28.456906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:28.456937 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:28.459646 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:28.460440 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:28.460445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:28.460448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:28.460591 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:28.460941 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:28.460944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:28.460946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:28.461252 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:28.461370 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:28.461966 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 06:38:28.462130 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:38:28.463315 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.445928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:32971 (local address 127.28.104.195:47433)
0619 06:38:28.446758 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:28.446824 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:28.448682 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:28.449412 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:28.449420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:28.449930 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:28.450535 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.450568 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.451610 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.451617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.452859 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.452875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.453686 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.453695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.453928 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.454545 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:38:28.454567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:28.456709 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:28.459773 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:28.459780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:28.459791 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:28.460332 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:28.460674 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:28.460678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:28.460680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:28.460852 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:28.461367 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:28.461372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:28.461661 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 06:38:28.461885 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 06:38:28.462260 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":321,"threads_started":1}
I20260619 06:38:28.464560 14343 heartbeater.cc:347] Connected to a master server at 127.28.104.254:32971
I20260619 06:38:28.464877 14343 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:28.465608 14343 heartbeater.cc:511] Master 127.28.104.254:32971 requested a full tablet report, sending...
I20260619 06:38:28.466921 13882 ts_manager.cc:194] Registered new tserver with Master: 0ea4b217fe3c4c299d6477766e68f68d (127.28.104.195:34371)
I20260619 06:38:28.467710 13882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:47433
I20260619 06:38:28.472508 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:38:28.489691 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.477125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.28.104.254:32971)
0619 06:38:28.477306 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:28.477313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:28.477771 (+ 458us) server_negotiation.cc:407] Connection header received
0619 06:38:28.477971 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:28.477977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:28.478050 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:28.478153 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:28.479503 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.480205 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.481600 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.481997 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.482130 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.484534 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:28.484539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:28.484542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:28.484571 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:28.487067 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:28.487708 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:28.487715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:28.487724 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:28.487844 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:28.488204 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:28.488212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:28.488216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:28.488527 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:28.488794 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:28.489131 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 06:38:28.489358 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:38:28.493952 13882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:38:28.500782 13882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:38:28.533402 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.514103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.28.104.194:35591)
0619 06:38:28.521388 (+ 7285us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:28.521401 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:28.522162 (+ 761us) server_negotiation.cc:407] Connection header received
0619 06:38:28.522481 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:28.522490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:28.522754 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:28.522991 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:28.524149 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.526400 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.530527 (+ 4127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.532044 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.532222 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.532407 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:28.532848 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 06:38:28.533049 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7066,"thread_start_us":6923,"threads_started":1}
I20260619 06:38:28.533663 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.514117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.28.104.193:36605)
0619 06:38:28.514894 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:28.514908 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:28.522765 (+ 7857us) server_negotiation.cc:407] Connection header received
0619 06:38:28.523076 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:28.523085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:28.523333 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:28.523523 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:28.524592 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.526638 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.531008 (+ 4370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.532461 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.532686 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.532866 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:28.533173 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 06:38:28.533329 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":353,"threads_started":1}
I20260619 06:38:28.533761 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.513553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36605 (local address 127.0.0.1:46816)
0619 06:38:28.522624 (+ 9071us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:28.522637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:28.522874 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:28.523897 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:28.523904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:28.523933 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:28.524395 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.524409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.526889 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.526898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.530776 (+ 3878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.530791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.532702 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.532709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.532891 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.532979 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 06:38:28.533130 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 06:38:28.533331 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8973,"mutex_wait_us":566,"thread_start_us":8044,"threads_started":1}
I20260619 06:38:28.534260 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.513453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:35591 (local address 127.0.0.1:48370)
0619 06:38:28.521995 (+ 8542us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:28.522033 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:28.522239 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:28.523320 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:28.523331 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:28.523371 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:28.523885 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.523902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.526665 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.526673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.530361 (+ 3688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.530379 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.532246 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.532252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.532487 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.532570 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 06:38:28.533715 (+ 1145us) client_negotiation.cc:240] Negotiation successful
0619 06:38:28.533847 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8272,"mutex_wait_us":28,"thread_start_us":8958,"threads_started":1}
I20260619 06:38:28.537241 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.522761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:34371 (local address 127.0.0.1:48972)
0619 06:38:28.529990 (+ 7229us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:28.530007 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:28.530116 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:28.530614 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:28.530620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:28.530648 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:28.531046 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.531055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.532481 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.532486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.534679 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.534695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.536480 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.536486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.536643 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.536681 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:38:28.536772 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:38:28.536899 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7105,"thread_start_us":6846,"threads_started":1}
I20260619 06:38:28.537349 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.523114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.28.104.195:34371)
0619 06:38:28.529538 (+ 6424us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:28.529549 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:28.530092 (+ 543us) server_negotiation.cc:407] Connection header received
0619 06:38:28.530229 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:28.530234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:28.530403 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:28.530623 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:28.531234 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.532263 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.534847 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.536273 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.536482 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.536642 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:28.536820 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 06:38:28.536977 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6261,"thread_start_us":6114,"threads_started":1}
I20260619 06:38:28.538043 14006 tablet_service.cc:1467] Processing CreateTablet for tablet a4e22e445667465a841141c09737cb18 (DEFAULT_TABLE table=test-table [id=1dbbe02e14834255a4c9b80e308ce3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:28.538041 14141 tablet_service.cc:1467] Processing CreateTablet for tablet a4e22e445667465a841141c09737cb18 (DEFAULT_TABLE table=test-table [id=1dbbe02e14834255a4c9b80e308ce3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:28.539562 14006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e22e445667465a841141c09737cb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:28.539564 14141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e22e445667465a841141c09737cb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:28.544909 14277 tablet_service.cc:1467] Processing CreateTablet for tablet a4e22e445667465a841141c09737cb18 (DEFAULT_TABLE table=test-table [id=1dbbe02e14834255a4c9b80e308ce3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:28.546701 14277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e22e445667465a841141c09737cb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:28.554986 14361 tablet_bootstrap.cc:492] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: Bootstrap starting.
I20260619 06:38:28.556828 14362 tablet_bootstrap.cc:492] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: Bootstrap starting.
I20260619 06:38:28.558981 14361 tablet_bootstrap.cc:654] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:28.560022 14361 log.cc:824] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:28.560341 14362 tablet_bootstrap.cc:654] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:28.561635 14362 log.cc:824] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:28.562601 14364 tablet_bootstrap.cc:492] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: Bootstrap starting.
I20260619 06:38:28.563254 14361 tablet_bootstrap.cc:492] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: No bootstrap required, opened a new log
I20260619 06:38:28.563526 14361 ts_tablet_manager.cc:1397] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:38:28.565989 14364 tablet_bootstrap.cc:654] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:28.567159 14364 log.cc:824] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:28.567224 14362 tablet_bootstrap.cc:492] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: No bootstrap required, opened a new log
I20260619 06:38:28.567516 14362 ts_tablet_manager.cc:1397] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260619 06:38:28.571614 14364 tablet_bootstrap.cc:492] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: No bootstrap required, opened a new log
I20260619 06:38:28.571869 14364 ts_tablet_manager.cc:1397] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260619 06:38:28.574218 14361 raft_consensus.cc:348] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a24827cc764d25bd377c1934bde6d7" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 35591 } } peers { permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 } } peers { permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 } }
I20260619 06:38:28.574899 14361 raft_consensus.cc:374] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:28.575043 14361 raft_consensus.cc:729] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a24827cc764d25bd377c1934bde6d7, State: Initialized, Role: FOLLOWER
I20260619 06:38:28.575563 14362 raft_consensus.cc:348] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a24827cc764d25bd377c1934bde6d7" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 35591 } } peers { permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 } } peers { permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 } }
I20260619 06:38:28.576040 14362 raft_consensus.cc:374] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:28.575893 14361 consensus_queue.cc:260] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a24827cc764d25bd377c1934bde6d7" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 35591 } } peers { permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 } } peers { permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 } }
I20260619 06:38:28.576176 14362 raft_consensus.cc:729] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886ceb12a99c4c73b286f9eb602e8361, State: Initialized, Role: FOLLOWER
I20260619 06:38:28.576747 14362 consensus_queue.cc:260] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a24827cc764d25bd377c1934bde6d7" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 35591 } } peers { permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 } } peers { permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 } }
I20260619 06:38:28.578301 14362 ts_tablet_manager.cc:1428] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260619 06:38:28.579221 14207 heartbeater.cc:503] Master 127.28.104.254:32971 was elected leader, sending a full tablet report...
I20260619 06:38:28.579910 14361 ts_tablet_manager.cc:1428] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260619 06:38:28.583875 14364 raft_consensus.cc:348] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a24827cc764d25bd377c1934bde6d7" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 35591 } } peers { permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 } } peers { permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 } }
I20260619 06:38:28.584295 14364 raft_consensus.cc:374] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:28.584403 14364 raft_consensus.cc:729] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea4b217fe3c4c299d6477766e68f68d, State: Initialized, Role: FOLLOWER
I20260619 06:38:28.585000 14364 consensus_queue.cc:260] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a24827cc764d25bd377c1934bde6d7" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 35591 } } peers { permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 } } peers { permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 } }
I20260619 06:38:28.585978 14343 heartbeater.cc:503] Master 127.28.104.254:32971 was elected leader, sending a full tablet report...
I20260619 06:38:28.586539 14364 ts_tablet_manager.cc:1428] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
W20260619 06:38:28.625484 14072 tablet.cc:2369] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:38:28.677332 14208 tablet.cc:2369] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:38:28.694201 14344 tablet.cc:2369] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:38:28.789467 14367 raft_consensus.cc:482] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:38:28.789806 14367 raft_consensus.cc:504] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a24827cc764d25bd377c1934bde6d7" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 35591 } } peers { permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 } } peers { permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 } }
I20260619 06:38:28.791646 14367 leader_election.cc:288] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 886ceb12a99c4c73b286f9eb602e8361 (127.28.104.193:36605), 0ea4b217fe3c4c299d6477766e68f68d (127.28.104.195:34371)
I20260619 06:38:28.798275 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.792085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:57613 (local address 127.28.104.195:34371)
0619 06:38:28.792361 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:28.792368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:28.792761 (+ 393us) server_negotiation.cc:407] Connection header received
0619 06:38:28.793063 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:28.793069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:28.793150 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:28.793230 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:28.793889 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.794627 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.796459 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.797280 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.797469 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.797544 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:28.797899 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 06:38:28.798022 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 06:38:28.798396 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.792183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:59715 (local address 127.28.104.193:36605)
0619 06:38:28.792435 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:28.792443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:28.793384 (+ 941us) server_negotiation.cc:407] Connection header received
0619 06:38:28.793556 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:28.793560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:28.793615 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:28.793683 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:28.794205 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.794915 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.796636 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.797471 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.797650 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.797734 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:28.798032 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 06:38:28.798154 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:38:28.798434 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.791957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:34371 (local address 127.28.104.194:57613)
0619 06:38:28.792648 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:28.792664 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:28.792856 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:28.793277 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:28.793283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:28.793316 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:28.793740 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.793750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.794757 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.794765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.796313 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.796326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.797494 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.797500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.797679 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.797723 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 06:38:28.797864 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:38:28.797988 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":46,"thread_start_us":343,"threads_started":1}
I20260619 06:38:28.798540 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.791957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:36605 (local address 127.28.104.194:59715)
0619 06:38:28.793277 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:28.793293 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:28.793431 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:28.793734 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:28.793738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:28.793762 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:28.794066 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.794073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.795057 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.795063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.796494 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:28.796505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.797680 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.797686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:28.797882 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.797905 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:38:28.798005 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:38:28.798127 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1189,"mutex_wait_us":117,"thread_start_us":939,"threads_started":1}
I20260619 06:38:28.799477 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e22e445667465a841141c09737cb18" candidate_uuid: "46a24827cc764d25bd377c1934bde6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea4b217fe3c4c299d6477766e68f68d" is_pre_election: true
I20260619 06:38:28.799484 14026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e22e445667465a841141c09737cb18" candidate_uuid: "46a24827cc764d25bd377c1934bde6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886ceb12a99c4c73b286f9eb602e8361" is_pre_election: true
I20260619 06:38:28.799947 14297 raft_consensus.cc:2454] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46a24827cc764d25bd377c1934bde6d7 in term 0.
I20260619 06:38:28.799961 14026 raft_consensus.cc:2454] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46a24827cc764d25bd377c1934bde6d7 in term 0.
I20260619 06:38:28.800619 14096 leader_election.cc:302] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ea4b217fe3c4c299d6477766e68f68d, 46a24827cc764d25bd377c1934bde6d7, 886ceb12a99c4c73b286f9eb602e8361; no voters:
I20260619 06:38:28.801007 14367 raft_consensus.cc:2781] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:38:28.801206 14367 raft_consensus.cc:482] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:38:28.801322 14367 raft_consensus.cc:3037] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:28.803557 14367 raft_consensus.cc:504] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a24827cc764d25bd377c1934bde6d7" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 35591 } } peers { permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 } } peers { permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 } }
I20260619 06:38:28.804180 14367 leader_election.cc:288] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [CANDIDATE]: Term 1 election: Requested vote from peers 886ceb12a99c4c73b286f9eb602e8361 (127.28.104.193:36605), 0ea4b217fe3c4c299d6477766e68f68d (127.28.104.195:34371)
I20260619 06:38:28.804576 14026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e22e445667465a841141c09737cb18" candidate_uuid: "46a24827cc764d25bd377c1934bde6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886ceb12a99c4c73b286f9eb602e8361"
I20260619 06:38:28.804672 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e22e445667465a841141c09737cb18" candidate_uuid: "46a24827cc764d25bd377c1934bde6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea4b217fe3c4c299d6477766e68f68d"
I20260619 06:38:28.804824 14026 raft_consensus.cc:3037] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:28.804895 14297 raft_consensus.cc:3037] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:28.808053 14297 raft_consensus.cc:2454] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46a24827cc764d25bd377c1934bde6d7 in term 1.
I20260619 06:38:28.808050 14026 raft_consensus.cc:2454] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46a24827cc764d25bd377c1934bde6d7 in term 1.
I20260619 06:38:28.808398 14096 leader_election.cc:302] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea4b217fe3c4c299d6477766e68f68d, 46a24827cc764d25bd377c1934bde6d7; no voters:
I20260619 06:38:28.808688 14367 raft_consensus.cc:2781] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:28.809274 14367 raft_consensus.cc:686] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 1 LEADER]: Becoming Leader. State: Replica: 46a24827cc764d25bd377c1934bde6d7, State: Running, Role: LEADER
I20260619 06:38:28.809815 14367 consensus_queue.cc:237] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a24827cc764d25bd377c1934bde6d7" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 35591 } } peers { permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 } } peers { permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 } }
I20260619 06:38:28.814076 13882 catalog_manager.cc:5495] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46a24827cc764d25bd377c1934bde6d7 (127.28.104.194). New cstate: current_term: 1 leader_uuid: "46a24827cc764d25bd377c1934bde6d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a24827cc764d25bd377c1934bde6d7" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 } health_report { overall_health: UNKNOWN } } }
I20260619 06:38:28.889302 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.883772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.28.104.194:35591)
0619 06:38:28.884002 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:28.884011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:28.884069 (+ 58us) server_negotiation.cc:407] Connection header received
0619 06:38:28.884247 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:28.884251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:28.884348 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:28.884537 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:38:28.884888 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.885725 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.887209 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.887578 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.887667 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.888086 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:38:28.888584 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:38:28.888697 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:28.888911 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 06:38:28.888973 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:38:28.897328 14026 raft_consensus.cc:1261] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361 [term 1 FOLLOWER]: Refusing update from remote peer 46a24827cc764d25bd377c1934bde6d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:28.897323 14297 raft_consensus.cc:1261] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d [term 1 FOLLOWER]: Refusing update from remote peer 46a24827cc764d25bd377c1934bde6d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:28.898276 14372 consensus_queue.cc:1035] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886ceb12a99c4c73b286f9eb602e8361" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:28.898599 14367 consensus_queue.cc:1035] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea4b217fe3c4c299d6477766e68f68d" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:28.910058 14375 mvcc.cc:205] Tried to move back new op lower bound from 7298462142033891328 to 7298462141688762368. Current Snapshot: MvccSnapshot[applied={T|T < 7298462142033891328}]
I20260619 06:38:28.922693 13881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:28.923005 13881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:28.926595 13881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 862c1845d28f4686b900dc5fc69ffcb2: Sending DeleteTablet for 3 replicas of tablet a4e22e445667465a841141c09737cb18
I20260619 06:38:28.927839 14141 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e22e445667465a841141c09737cb18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:28 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:48370
I20260619 06:38:28.927944 14006 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e22e445667465a841141c09737cb18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:28 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:46816
I20260619 06:38:28.929045 14277 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e22e445667465a841141c09737cb18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:28 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:48972
I20260619 06:38:28.929615 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 13943
I20260619 06:38:28.933482 14387 tablet_replica.cc:330] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: stopping tablet replica
I20260619 06:38:28.933872 14387 raft_consensus.cc:2229] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:28.934311 14387 raft_consensus.cc:2258] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:28.941009 14387 ts_tablet_manager.cc:1905] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:28.945514 14386 tablet_replica.cc:330] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: stopping tablet replica
I20260619 06:38:28.946053 14386 raft_consensus.cc:2229] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:28.946656 14386 raft_consensus.cc:2258] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:28.951392 14385 tablet_replica.cc:330] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: stopping tablet replica
I20260619 06:38:28.951778 14385 raft_consensus.cc:2229] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:38:28.952365 14385 raft_consensus.cc:2258] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:28.952970 14386 ts_tablet_manager.cc:1905] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:28.953836 14387 ts_tablet_manager.cc:1918] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:28.954011 14387 log.cc:1196] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/a4e22e445667465a841141c09737cb18
I20260619 06:38:28.954447 14387 ts_tablet_manager.cc:1939] T a4e22e445667465a841141c09737cb18 P 0ea4b217fe3c4c299d6477766e68f68d: Deleting consensus metadata
I20260619 06:38:28.956048 13869 catalog_manager.cc:4841] TS 0ea4b217fe3c4c299d6477766e68f68d (127.28.104.195:34371): tablet a4e22e445667465a841141c09737cb18 (table test-table [id=1dbbe02e14834255a4c9b80e308ce3c0]) successfully deleted
I20260619 06:38:28.957402 14385 ts_tablet_manager.cc:1905] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 06:38:28 dist-test-slave-nf4n krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:38:28.960304 14386 ts_tablet_manager.cc:1918] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:28.960505 14386 log.cc:1196] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/a4e22e445667465a841141c09737cb18
I20260619 06:38:28.961115 14386 ts_tablet_manager.cc:1939] T a4e22e445667465a841141c09737cb18 P 886ceb12a99c4c73b286f9eb602e8361: Deleting consensus metadata
I20260619 06:38:28.963121 13868 catalog_manager.cc:4841] TS 886ceb12a99c4c73b286f9eb602e8361 (127.28.104.193:36605): tablet a4e22e445667465a841141c09737cb18 (table test-table [id=1dbbe02e14834255a4c9b80e308ce3c0]) successfully deleted
I20260619 06:38:28.965772 14385 ts_tablet_manager.cc:1918] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:28.966013 14385 log.cc:1196] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/a4e22e445667465a841141c09737cb18
I20260619 06:38:28.966482 14385 ts_tablet_manager.cc:1939] T a4e22e445667465a841141c09737cb18 P 46a24827cc764d25bd377c1934bde6d7: Deleting consensus metadata
I20260619 06:38:28.968202 13869 catalog_manager.cc:4841] TS 46a24827cc764d25bd377c1934bde6d7 (127.28.104.194:35591): tablet a4e22e445667465a841141c09737cb18 (table test-table [id=1dbbe02e14834255a4c9b80e308ce3c0]) successfully deleted
I20260619 06:38:28.969192 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:28.930675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.28.104.193:36605)
0619 06:38:28.930891 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:28.930900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:28.934995 (+ 4095us) server_negotiation.cc:407] Connection header received
0619 06:38:28.939873 (+ 4878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:28.939880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:28.939961 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:28.940029 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:28.941775 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.943007 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.945327 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:28.946000 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:28.946109 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:28.961760 (+ 15651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:28.961772 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:28.961788 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:28.961843 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:28.965762 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:28.966707 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:28.966715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:28.966718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:28.966924 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:28.967398 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:28.967404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:28.967408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:28.967854 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:28.968159 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:28.968525 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 06:38:28.968769 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:38:28.970342 14066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:29.073807 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 13943
I20260619 06:38:29.093885 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 14077
Jun 19 06:38:29 dist-test-slave-nf4n krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:38:29.110945 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:29.094755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.28.104.194:35591)
0619 06:38:29.094946 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:29.094954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:29.094980 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:38:29.095157 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:29.095164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:29.095280 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:29.095398 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:29.096883 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:29.098128 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:29.099469 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:29.100176 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:29.100311 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:29.104912 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:29.104927 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:29.104947 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:29.104990 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:29.108065 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:29.108902 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:29.108946 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:29.108949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:29.109147 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:29.109545 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:29.109548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:29.109550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:29.109890 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:29.110173 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:29.110465 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:38:29.110622 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:38:29.111799 14202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:29.225052 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 14077
I20260619 06:38:29.243459 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 14213
Jun 19 06:38:29 dist-test-slave-nf4n krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:38:29.258984 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:29.244344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.28.104.195:34371)
0619 06:38:29.244562 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:29.244569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:29.244592 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:38:29.244693 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:29.244699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:29.244793 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:29.244900 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:29.246812 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:29.247727 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:29.248931 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:29.249400 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:29.249493 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:29.253706 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:29.253720 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:29.253736 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:29.253769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:29.256497 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:29.257211 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:29.257222 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:29.257225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:29.257380 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:29.257692 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:29.257695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:29.257697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:29.258031 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:29.258236 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:29.258465 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 06:38:29.258628 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:38:29.259625 14338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:29.348625 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 14213
I20260619 06:38:29.365147 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 13850
I20260619 06:38:29.366358 13912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:29.478325 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 13850
2026-06-19T06:38:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4514 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T06:38:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:29Z Disabled control of system clock
I20260619 06:38:29.556993 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:36765
--webserver_interface=127.28.104.254
--webserver_port=0
--builtin_ntp_servers=127.28.104.212:39201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:36765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:38:29.892367 14396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:29.892671 14396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:29.892774 14396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:29.900079 14396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:29.900182 14396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:29.900233 14396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:29.900287 14396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:29.900323 14396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:29.910254 14396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:36765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:36765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:29.911473 14396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:29.913316 14396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:29.925093 14402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:29.925060 14404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:29.925060 14401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:29.925894 14396 server_base.cc:1034] running on GCE node
I20260619 06:38:29.926697 14396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:29.928081 14396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:29.929322 14396 hybrid_clock.cc:648] HybridClock initialized: now 1781851109929274 us; error 56 us; skew 500 ppm
I20260619 06:38:29.929754 14396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:29.937517 14396 webserver.cc:460] Webserver started at http://127.28.104.254:43949/ using document root <none> and password file <none>
I20260619 06:38:29.938186 14396 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:29.938341 14396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:29.938647 14396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:29.941591 14396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "ca88a50bd11b4d97819445ba3411c821"
format_stamp: "Formatted at 2026-06-19 06:38:29 on dist-test-slave-nf4n"
server_key: "41065f30ccb51fc62733e36f59d1c3ec"
server_key_iv: "dc78f93ff46f5ac8fe07bc3c6a11cbec"
server_key_version: "encryptionkey@0"
I20260619 06:38:29.942345 14396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "ca88a50bd11b4d97819445ba3411c821"
format_stamp: "Formatted at 2026-06-19 06:38:29 on dist-test-slave-nf4n"
server_key: "41065f30ccb51fc62733e36f59d1c3ec"
server_key_iv: "dc78f93ff46f5ac8fe07bc3c6a11cbec"
server_key_version: "encryptionkey@0"
I20260619 06:38:29.947221 14396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 06:38:29.950778 14410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:29.952097 14396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:38:29.952237 14396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "ca88a50bd11b4d97819445ba3411c821"
format_stamp: "Formatted at 2026-06-19 06:38:29 on dist-test-slave-nf4n"
server_key: "41065f30ccb51fc62733e36f59d1c3ec"
server_key_iv: "dc78f93ff46f5ac8fe07bc3c6a11cbec"
server_key_version: "encryptionkey@0"
I20260619 06:38:29.952433 14396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:29.975678 14396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:29.976467 14396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:29.976704 14396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:29.994812 14396 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:36765
I20260619 06:38:29.994854 14461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:36765 every 8 connection(s)
I20260619 06:38:29.996234 14396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:29.999564 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 14396
I20260619 06:38:29.999737 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:30.000072 14462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:30.000231 29091 external_mini_cluster.cc:1428] Setting key 6b2c751ae69f35ec0d19c94573fbe9c6
I20260619 06:38:30.015442 14462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821: Bootstrap starting.
I20260619 06:38:30.015463 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:30.002135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.28.104.254:36765)
0619 06:38:30.003187 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:30.003206 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:30.003254 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:38:30.005278 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:30.005308 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:30.005910 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:30.006353 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:30.006892 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:30.008889 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:30.010312 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:30.011349 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:30.011546 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:30.012639 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:30.012654 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:30.012677 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:30.013449 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:30.013541 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:30.013565 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:30.013671 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:30.013869 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 06:38:30.014198 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":304,"threads_started":1}
I20260619 06:38:30.018775 14462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:30.019672 14462 log.cc:824] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:30.022459 14462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821: No bootstrap required, opened a new log
I20260619 06:38:30.029819 14462 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88a50bd11b4d97819445ba3411c821" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36765 } }
I20260619 06:38:30.030259 14462 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:30.030336 14462 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca88a50bd11b4d97819445ba3411c821, State: Initialized, Role: FOLLOWER
I20260619 06:38:30.030819 14462 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88a50bd11b4d97819445ba3411c821" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36765 } }
I20260619 06:38:30.031014 14462 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:30.031104 14462 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:30.031216 14462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:30.033340 14462 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88a50bd11b4d97819445ba3411c821" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36765 } }
I20260619 06:38:30.033779 14462 leader_election.cc:302] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca88a50bd11b4d97819445ba3411c821; no voters:
I20260619 06:38:30.034296 14462 leader_election.cc:288] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:30.034536 14468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:30.035262 14468 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [term 1 LEADER]: Becoming Leader. State: Replica: ca88a50bd11b4d97819445ba3411c821, State: Running, Role: LEADER
I20260619 06:38:30.035930 14468 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88a50bd11b4d97819445ba3411c821" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36765 } }
I20260619 06:38:30.036532 14462 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:30.038707 14470 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca88a50bd11b4d97819445ba3411c821. Latest consensus state: current_term: 1 leader_uuid: "ca88a50bd11b4d97819445ba3411c821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88a50bd11b4d97819445ba3411c821" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36765 } } }
I20260619 06:38:30.038662 14469 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca88a50bd11b4d97819445ba3411c821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88a50bd11b4d97819445ba3411c821" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 36765 } } }
I20260619 06:38:30.039083 14469 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:30.039080 14470 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:30.041724 14475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:30.047029 14475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:30.055586 14475 catalog_manager.cc:1347] Generated new cluster ID: b6f30c79e9f5452182fb6b5769f2edf5
I20260619 06:38:30.055697 14475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:30.065099 14475 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:30.066895 14475 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:30.075017 14475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821: Generated new TSK 0
I20260619 06:38:30.076004 14475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:38:30.086052 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36765
--builtin_ntp_servers=127.28.104.212:39201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:38:30.415627 14487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:30.415918 14487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:30.416006 14487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:30.423218 14487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:30.423331 14487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:30.423462 14487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:38:30.432533 14487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:30.433948 14487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:30.435626 14487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:30.448729 14495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:30.448832 14493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:30.448846 14492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:30.449874 14487 server_base.cc:1034] running on GCE node
I20260619 06:38:30.450469 14487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:30.451388 14487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:30.452615 14487 hybrid_clock.cc:648] HybridClock initialized: now 1781851110452573 us; error 51 us; skew 500 ppm
I20260619 06:38:30.452921 14487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:30.455150 14487 webserver.cc:460] Webserver started at http://127.28.104.193:43013/ using document root <none> and password file <none>
I20260619 06:38:30.455926 14487 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:30.456054 14487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:30.456372 14487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:30.459280 14487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "fa3fa622e01246a2afbf1f2e8cd691ab"
format_stamp: "Formatted at 2026-06-19 06:38:30 on dist-test-slave-nf4n"
server_key: "a52d8ea1168220a1002e89cf4b3ff64d"
server_key_iv: "851824d5fbf04551a654606acc244d43"
server_key_version: "encryptionkey@0"
I20260619 06:38:30.460035 14487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "fa3fa622e01246a2afbf1f2e8cd691ab"
format_stamp: "Formatted at 2026-06-19 06:38:30 on dist-test-slave-nf4n"
server_key: "a52d8ea1168220a1002e89cf4b3ff64d"
server_key_iv: "851824d5fbf04551a654606acc244d43"
server_key_version: "encryptionkey@0"
I20260619 06:38:30.464846 14487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260619 06:38:30.468117 14501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:30.469445 14487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 06:38:30.469621 14487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "fa3fa622e01246a2afbf1f2e8cd691ab"
format_stamp: "Formatted at 2026-06-19 06:38:30 on dist-test-slave-nf4n"
server_key: "a52d8ea1168220a1002e89cf4b3ff64d"
server_key_iv: "851824d5fbf04551a654606acc244d43"
server_key_version: "encryptionkey@0"
I20260619 06:38:30.469835 14487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:30.486990 14487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:30.487845 14487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:30.488157 14487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:30.489168 14487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:30.490621 14487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:30.490742 14487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:30.490883 14487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:30.490971 14487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:30.523564 14487 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:43903
I20260619 06:38:30.523650 14613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:43903 every 8 connection(s)
I20260619 06:38:30.524905 14487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:38:30.532582 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 14487
I20260619 06:38:30.532786 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:38:30.533320 29091 external_mini_cluster.cc:1428] Setting key 8f07a48b3ca80a8b2a04a3e56115dc67
I20260619 06:38:30.536088 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36765
--builtin_ntp_servers=127.28.104.212:39201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:38:30.538690 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:30.527453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:40691 (local address 127.28.104.254:36765)
0619 06:38:30.527900 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:30.527906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:30.529474 (+ 1568us) server_negotiation.cc:407] Connection header received
0619 06:38:30.531215 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:30.531219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:30.531283 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:30.531375 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:30.532479 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:30.533511 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:30.535685 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:30.536112 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:30.536270 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:30.537934 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:30.537938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:30.537942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:30.537958 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:30.538003 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:30.538015 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:30.538104 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:30.538388 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 06:38:30.538461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":204,"threads_started":1}
I20260619 06:38:30.540131 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:30.528120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:36765 (local address 127.28.104.193:40691)
0619 06:38:30.529326 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:30.529364 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:30.530968 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:30.531568 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:30.531576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:30.531666 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:30.532287 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:30.532306 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:30.533670 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:30.533680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:30.535532 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:30.535544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:30.536358 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:30.536367 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:30.536731 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:30.537564 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:38:30.537570 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:30.537624 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:38:30.537731 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:30.538131 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:30.538144 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:30.538155 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:38:30.538373 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 06:38:30.538787 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":251,"threads_started":1}
I20260619 06:38:30.541815 14615 heartbeater.cc:347] Connected to a master server at 127.28.104.254:36765
I20260619 06:38:30.542122 14615 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:30.542881 14615 heartbeater.cc:511] Master 127.28.104.254:36765 requested a full tablet report, sending...
I20260619 06:38:30.545076 14427 ts_manager.cc:194] Registered new tserver with Master: fa3fa622e01246a2afbf1f2e8cd691ab (127.28.104.193:43903)
I20260619 06:38:30.546706 14427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.193:40691
W20260619 06:38:30.862684 14619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:30.862972 14619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:30.863109 14619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:30.870314 14619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:30.870419 14619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:30.870591 14619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:38:30.879822 14619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:30.881160 14619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:30.882915 14619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:30.895875 14627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:30.895835 14625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:30.895911 14624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:30.896834 14619 server_base.cc:1034] running on GCE node
I20260619 06:38:30.897459 14619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:30.898414 14619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:30.899639 14619 hybrid_clock.cc:648] HybridClock initialized: now 1781851110899594 us; error 57 us; skew 500 ppm
I20260619 06:38:30.899960 14619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:30.902107 14619 webserver.cc:460] Webserver started at http://127.28.104.194:34613/ using document root <none> and password file <none>
I20260619 06:38:30.902760 14619 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:30.902887 14619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:30.903191 14619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:30.906064 14619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "05dead650d6441a98d0de9b5844bb645"
format_stamp: "Formatted at 2026-06-19 06:38:30 on dist-test-slave-nf4n"
server_key: "89bde3f9746c43d3ccdcd8f84f1d4d5b"
server_key_iv: "6e665a12fbb6fa2cfc7ee08fc828b1c8"
server_key_version: "encryptionkey@0"
I20260619 06:38:30.906840 14619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "05dead650d6441a98d0de9b5844bb645"
format_stamp: "Formatted at 2026-06-19 06:38:30 on dist-test-slave-nf4n"
server_key: "89bde3f9746c43d3ccdcd8f84f1d4d5b"
server_key_iv: "6e665a12fbb6fa2cfc7ee08fc828b1c8"
server_key_version: "encryptionkey@0"
I20260619 06:38:30.911942 14619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:38:30.915258 14633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:30.916543 14619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:38:30.916706 14619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "05dead650d6441a98d0de9b5844bb645"
format_stamp: "Formatted at 2026-06-19 06:38:30 on dist-test-slave-nf4n"
server_key: "89bde3f9746c43d3ccdcd8f84f1d4d5b"
server_key_iv: "6e665a12fbb6fa2cfc7ee08fc828b1c8"
server_key_version: "encryptionkey@0"
I20260619 06:38:30.916895 14619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:30.938552 14619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:30.939365 14619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:30.939657 14619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:30.940591 14619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:30.942166 14619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:30.942281 14619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:30.942399 14619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:30.942485 14619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:30.975203 14619 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:34193
I20260619 06:38:30.975292 14745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:34193 every 8 connection(s)
I20260619 06:38:30.976559 14619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:38:30.982662 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 14619
I20260619 06:38:30.982851 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:38:30.983304 29091 external_mini_cluster.cc:1428] Setting key a397c9d35e4669f9e6f6f2d265376771
I20260619 06:38:30.986517 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36765
--builtin_ntp_servers=127.28.104.212:39201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:38:30.991285 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:30.979218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:37181 (local address 127.28.104.254:36765)
0619 06:38:30.979392 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:30.979397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:30.980883 (+ 1486us) server_negotiation.cc:407] Connection header received
0619 06:38:30.983365 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:30.983371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:30.983475 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:30.983615 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:30.984979 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:30.985710 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:30.988373 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:30.988771 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:30.988891 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:30.990588 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:30.990594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:30.990597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:30.990614 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:30.990645 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:30.990657 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:30.990729 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:30.990996 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 06:38:30.991070 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:38:30.993417 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:30.979764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:36765 (local address 127.28.104.194:37181)
0619 06:38:30.980737 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:30.980781 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:30.983075 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:30.983867 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:30.983875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:30.983974 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:30.984759 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:30.984787 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:30.985889 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:30.985895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:30.988220 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:30.988241 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:30.989049 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:30.989060 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:30.989404 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:30.990214 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:38:30.990220 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:30.990273 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:38:30.990382 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:30.990761 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:30.990775 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:30.990786 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:38:30.990993 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 06:38:30.991438 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":261,"threads_started":1}
I20260619 06:38:30.994899 14746 heartbeater.cc:347] Connected to a master server at 127.28.104.254:36765
I20260619 06:38:30.995282 14746 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:30.996292 14746 heartbeater.cc:511] Master 127.28.104.254:36765 requested a full tablet report, sending...
I20260619 06:38:30.998059 14427 ts_manager.cc:194] Registered new tserver with Master: 05dead650d6441a98d0de9b5844bb645 (127.28.104.194:34193)
I20260619 06:38:30.999249 14427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.194:37181
W20260619 06:38:31.311347 14749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:31.311679 14749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:31.311843 14749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:31.319025 14749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:31.319155 14749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:31.319335 14749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:38:31.328271 14749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:39201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:36765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:31.329571 14749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:31.331323 14749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:31.344218 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:31.344192 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:31.344400 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:31.344969 14749 server_base.cc:1034] running on GCE node
I20260619 06:38:31.345592 14749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:31.346482 14749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:31.347725 14749 hybrid_clock.cc:648] HybridClock initialized: now 1781851111347664 us; error 74 us; skew 500 ppm
I20260619 06:38:31.348039 14749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:31.350350 14749 webserver.cc:460] Webserver started at http://127.28.104.195:38677/ using document root <none> and password file <none>
I20260619 06:38:31.351022 14749 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:31.351150 14749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:31.351460 14749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:31.354341 14749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "c6213c1dff654d2f9e08d9acce6da51f"
format_stamp: "Formatted at 2026-06-19 06:38:31 on dist-test-slave-nf4n"
server_key: "c1da29e012addd9cdbddbf5dcf1e77f2"
server_key_iv: "68f7d97567881d60b60cb554f8dc49f4"
server_key_version: "encryptionkey@0"
I20260619 06:38:31.355136 14749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "c6213c1dff654d2f9e08d9acce6da51f"
format_stamp: "Formatted at 2026-06-19 06:38:31 on dist-test-slave-nf4n"
server_key: "c1da29e012addd9cdbddbf5dcf1e77f2"
server_key_iv: "68f7d97567881d60b60cb554f8dc49f4"
server_key_version: "encryptionkey@0"
I20260619 06:38:31.360358 14749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:38:31.363754 14764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:31.365062 14749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:38:31.365253 14749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "c6213c1dff654d2f9e08d9acce6da51f"
format_stamp: "Formatted at 2026-06-19 06:38:31 on dist-test-slave-nf4n"
server_key: "c1da29e012addd9cdbddbf5dcf1e77f2"
server_key_iv: "68f7d97567881d60b60cb554f8dc49f4"
server_key_version: "encryptionkey@0"
I20260619 06:38:31.365468 14749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:31.380982 14749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:31.381820 14749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:31.382113 14749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:31.383051 14749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:31.384544 14749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:31.384665 14749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:31.384799 14749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:31.384886 14749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:31.417641 14749 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:34467
I20260619 06:38:31.417695 14876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:34467 every 8 connection(s)
I20260619 06:38:31.418958 14749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:38:31.424461 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 14749
I20260619 06:38:31.424645 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:38:31.425119 29091 external_mini_cluster.cc:1428] Setting key ebf003ca3887f7b6f1f79577e5345dd8
I20260619 06:38:31.431638 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.421783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:52925 (local address 127.28.104.254:36765)
0619 06:38:31.421972 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.421976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.423373 (+ 1397us) server_negotiation.cc:407] Connection header received
0619 06:38:31.425339 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.425344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.425442 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.425558 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:31.426616 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.427357 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.428875 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.429334 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.429470 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.430811 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:31.430816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:31.430819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:31.430837 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:31.430867 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:31.430880 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:31.430977 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.431201 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.431294 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:38:31.432539 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.422264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:36765 (local address 127.28.104.195:52925)
0619 06:38:31.423217 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:31.423258 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:31.424966 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:31.425810 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:31.425819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:31.425906 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:31.426445 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.426460 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.427509 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.427515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.428727 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.428739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.429548 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.429554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.429789 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.430475 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:38:31.430478 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:31.430513 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:38:31.430583 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:31.430986 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:31.430999 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:31.431006 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:38:31.431183 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 06:38:31.431539 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":296,"threads_started":1}
I20260619 06:38:31.433810 14877 heartbeater.cc:347] Connected to a master server at 127.28.104.254:36765
I20260619 06:38:31.434051 14877 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:31.434836 14877 heartbeater.cc:511] Master 127.28.104.254:36765 requested a full tablet report, sending...
I20260619 06:38:31.436091 14427 ts_manager.cc:194] Registered new tserver with Master: c6213c1dff654d2f9e08d9acce6da51f (127.28.104.195:34467)
I20260619 06:38:31.436934 14427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.195:52925
I20260619 06:38:31.439742 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:38:31.449115 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.443699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.28.104.254:36765)
0619 06:38:31.443912 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.443920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.444314 (+ 394us) server_negotiation.cc:407] Connection header received
0619 06:38:31.444454 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.444458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.444522 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.444597 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:31.444977 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.445701 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.447038 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.447420 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.447522 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.448396 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:31.448401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:31.448409 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:31.448426 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:31.448456 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:31.448468 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:31.448537 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.448653 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.448721 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:38:31.452630 14427 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:38:31.459656 14427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:38:31.482800 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.471825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.28.104.194:34193)
0619 06:38:31.472422 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.472432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.472466 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:38:31.472675 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.472685 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.472973 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.474030 (+ 1057us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:31.474350 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.476415 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.479563 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.481130 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.481351 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.481607 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.482249 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.482431 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":244,"threads_started":1}
I20260619 06:38:31.482877 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.471595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:34193 (local address 127.0.0.1:35426)
0619 06:38:31.472290 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:31.472326 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:31.472472 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:31.473312 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:31.473319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:31.473354 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:31.473808 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.473825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.476588 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.476597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.479374 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.479388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.481650 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.481660 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.481943 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.482030 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 06:38:31.482200 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 06:38:31.482387 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":27,"thread_start_us":260,"threads_started":1}
I20260619 06:38:31.485915 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.472782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:34467 (local address 127.0.0.1:45868)
0619 06:38:31.473454 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:31.473477 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:31.473620 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:31.478532 (+ 4912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:31.478541 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:31.478599 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:31.479099 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.479110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.481063 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.481074 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.483222 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.483237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.484991 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.485003 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.485244 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.485283 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:38:31.485386 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:38:31.485520 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":59,"spinlock_wait_cycles":4352,"thread_start_us":270,"threads_started":1}
I20260619 06:38:31.485992 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.473003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.28.104.195:34467)
0619 06:38:31.477929 (+ 4926us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.477937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.477977 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:38:31.478078 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.478084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.478320 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.478537 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:31.479283 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.480831 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.483655 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.484784 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.484943 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.485160 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.485459 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.485623 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4737,"thread_start_us":4527,"threads_started":1}
I20260619 06:38:31.487241 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.473749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.28.104.193:43903)
0619 06:38:31.474485 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.474497 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.477807 (+ 3310us) server_negotiation.cc:407] Connection header received
0619 06:38:31.477914 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.477922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.478165 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.479123 (+ 958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:31.479350 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.480614 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.483752 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.486171 (+ 2419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.486341 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.486534 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.486736 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.486888 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"spinlock_wait_cycles":96640,"thread_start_us":450,"threads_started":1}
I20260619 06:38:31.487272 14681 tablet_service.cc:1467] Processing CreateTablet for tablet b6ec03e628d049ffa8cdb90c9c6504e4 (DEFAULT_TABLE table=test-table [id=857c3064b25f48ada75d758617ff07e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:31.488941 14681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ec03e628d049ffa8cdb90c9c6504e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:31.489607 14812 tablet_service.cc:1467] Processing CreateTablet for tablet b6ec03e628d049ffa8cdb90c9c6504e4 (DEFAULT_TABLE table=test-table [id=857c3064b25f48ada75d758617ff07e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:31.490684 14812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ec03e628d049ffa8cdb90c9c6504e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:31.493126 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.474079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:43903 (local address 127.0.0.1:43406)
0619 06:38:31.477188 (+ 3109us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:31.477207 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:31.477345 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:31.478382 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:31.478387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:31.478412 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:31.478902 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.478920 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.480754 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.480760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.483482 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.483496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.486352 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.486358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.486534 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.486572 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:38:31.486681 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 06:38:31.486811 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2966,"mutex_wait_us":304,"thread_start_us":2770,"threads_started":1}
I20260619 06:38:31.497396 14549 tablet_service.cc:1467] Processing CreateTablet for tablet b6ec03e628d049ffa8cdb90c9c6504e4 (DEFAULT_TABLE table=test-table [id=857c3064b25f48ada75d758617ff07e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:31.498775 14549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ec03e628d049ffa8cdb90c9c6504e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:31.502506 14895 tablet_bootstrap.cc:492] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: Bootstrap starting.
I20260619 06:38:31.505856 14896 tablet_bootstrap.cc:492] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: Bootstrap starting.
I20260619 06:38:31.505889 14895 tablet_bootstrap.cc:654] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:31.507664 14895 log.cc:824] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:31.509625 14896 tablet_bootstrap.cc:654] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:31.510923 14896 log.cc:824] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:31.511018 14895 tablet_bootstrap.cc:492] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: No bootstrap required, opened a new log
I20260619 06:38:31.511296 14895 ts_tablet_manager.cc:1397] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 06:38:31.512701 14899 tablet_bootstrap.cc:492] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: Bootstrap starting.
I20260619 06:38:31.515609 14896 tablet_bootstrap.cc:492] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: No bootstrap required, opened a new log
I20260619 06:38:31.515974 14896 ts_tablet_manager.cc:1397] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20260619 06:38:31.515986 14899 tablet_bootstrap.cc:654] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:31.516924 14899 log.cc:824] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:31.519253 14899 tablet_bootstrap.cc:492] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: No bootstrap required, opened a new log
I20260619 06:38:31.519512 14899 ts_tablet_manager.cc:1397] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 06:38:31.523203 14895 raft_consensus.cc:348] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 } } peers { permanent_uuid: "c6213c1dff654d2f9e08d9acce6da51f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34467 } } peers { permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 } }
I20260619 06:38:31.523697 14895 raft_consensus.cc:374] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:31.523833 14895 raft_consensus.cc:729] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05dead650d6441a98d0de9b5844bb645, State: Initialized, Role: FOLLOWER
I20260619 06:38:31.524493 14895 consensus_queue.cc:260] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 } } peers { permanent_uuid: "c6213c1dff654d2f9e08d9acce6da51f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34467 } } peers { permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 } }
I20260619 06:38:31.526332 14895 ts_tablet_manager.cc:1428] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260619 06:38:31.526639 14746 heartbeater.cc:503] Master 127.28.104.254:36765 was elected leader, sending a full tablet report...
I20260619 06:38:31.529772 14896 raft_consensus.cc:348] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 } } peers { permanent_uuid: "c6213c1dff654d2f9e08d9acce6da51f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34467 } } peers { permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 } }
I20260619 06:38:31.530205 14896 raft_consensus.cc:374] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:31.530298 14896 raft_consensus.cc:729] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6213c1dff654d2f9e08d9acce6da51f, State: Initialized, Role: FOLLOWER
I20260619 06:38:31.530874 14896 consensus_queue.cc:260] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 } } peers { permanent_uuid: "c6213c1dff654d2f9e08d9acce6da51f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34467 } } peers { permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 } }
I20260619 06:38:31.532155 14899 raft_consensus.cc:348] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 } } peers { permanent_uuid: "c6213c1dff654d2f9e08d9acce6da51f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34467 } } peers { permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 } }
I20260619 06:38:31.532660 14899 raft_consensus.cc:374] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:31.532855 14899 raft_consensus.cc:729] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3fa622e01246a2afbf1f2e8cd691ab, State: Initialized, Role: FOLLOWER
I20260619 06:38:31.533640 14899 consensus_queue.cc:260] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 } } peers { permanent_uuid: "c6213c1dff654d2f9e08d9acce6da51f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34467 } } peers { permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 } }
I20260619 06:38:31.534271 14877 heartbeater.cc:503] Master 127.28.104.254:36765 was elected leader, sending a full tablet report...
I20260619 06:38:31.534935 14896 ts_tablet_manager.cc:1428] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20260619 06:38:31.535744 14615 heartbeater.cc:503] Master 127.28.104.254:36765 was elected leader, sending a full tablet report...
I20260619 06:38:31.536350 14899 ts_tablet_manager.cc:1428] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: Time spent starting tablet: real 0.017s user 0.005s sys 0.009s
I20260619 06:38:31.570245 14902 raft_consensus.cc:482] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:38:31.570518 14902 raft_consensus.cc:504] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 } } peers { permanent_uuid: "c6213c1dff654d2f9e08d9acce6da51f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34467 } } peers { permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 } }
I20260619 06:38:31.572137 14902 leader_election.cc:288] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05dead650d6441a98d0de9b5844bb645 (127.28.104.194:34193), fa3fa622e01246a2afbf1f2e8cd691ab (127.28.104.193:43903)
I20260619 06:38:31.578974 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.572789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:37559 (local address 127.28.104.194:34193)
0619 06:38:31.573060 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.573067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.573089 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:38:31.573171 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.573179 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.573270 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.573386 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:31.573937 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.574880 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.576937 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.578153 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.578306 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.578357 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.578690 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.578787 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:38:31.579144 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.572601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:34193 (local address 127.28.104.195:37559)
0619 06:38:31.572813 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:31.572835 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:31.572988 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:31.573419 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:31.573424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:31.573457 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:31.573754 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.573768 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.575007 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.575010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.576773 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.576785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.578325 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.578335 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.578513 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.578549 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:38:31.578646 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:38:31.578835 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 06:38:31.580255 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.573228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:42995 (local address 127.28.104.193:43903)
0619 06:38:31.573440 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.573446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.574098 (+ 652us) server_negotiation.cc:407] Connection header received
0619 06:38:31.574181 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.574188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.574269 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.574377 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:31.574944 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.575842 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.578182 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.579436 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.579600 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.579665 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.579901 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.580025 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:38:31.580297 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.573044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:43903 (local address 127.28.104.195:42995)
0619 06:38:31.573996 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:31.574009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:31.574097 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:31.574424 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:31.574428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:31.574453 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:31.574768 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.574776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.576003 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.576008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.577967 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:31.577981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.579589 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.579593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:31.579712 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.579735 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:38:31.579870 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 06:38:31.579992 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":586,"threads_started":1}
I20260619 06:38:31.580168 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ec03e628d049ffa8cdb90c9c6504e4" candidate_uuid: "c6213c1dff654d2f9e08d9acce6da51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05dead650d6441a98d0de9b5844bb645" is_pre_election: true
I20260619 06:38:31.580749 14701 raft_consensus.cc:2454] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6213c1dff654d2f9e08d9acce6da51f in term 0.
I20260619 06:38:31.581521 14765 leader_election.cc:302] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05dead650d6441a98d0de9b5844bb645, c6213c1dff654d2f9e08d9acce6da51f; no voters:
I20260619 06:38:31.581523 14569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ec03e628d049ffa8cdb90c9c6504e4" candidate_uuid: "c6213c1dff654d2f9e08d9acce6da51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" is_pre_election: true
I20260619 06:38:31.581967 14902 raft_consensus.cc:2781] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:38:31.582084 14902 raft_consensus.cc:482] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:38:31.582100 14569 raft_consensus.cc:2454] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6213c1dff654d2f9e08d9acce6da51f in term 0.
I20260619 06:38:31.582224 14902 raft_consensus.cc:3037] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:31.584452 14902 raft_consensus.cc:504] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 } } peers { permanent_uuid: "c6213c1dff654d2f9e08d9acce6da51f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34467 } } peers { permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 } }
I20260619 06:38:31.584985 14902 leader_election.cc:288] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [CANDIDATE]: Term 1 election: Requested vote from peers 05dead650d6441a98d0de9b5844bb645 (127.28.104.194:34193), fa3fa622e01246a2afbf1f2e8cd691ab (127.28.104.193:43903)
I20260619 06:38:31.585659 14569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ec03e628d049ffa8cdb90c9c6504e4" candidate_uuid: "c6213c1dff654d2f9e08d9acce6da51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab"
I20260619 06:38:31.585642 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ec03e628d049ffa8cdb90c9c6504e4" candidate_uuid: "c6213c1dff654d2f9e08d9acce6da51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05dead650d6441a98d0de9b5844bb645"
I20260619 06:38:31.585834 14569 raft_consensus.cc:3037] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:31.585863 14701 raft_consensus.cc:3037] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:31.587947 14569 raft_consensus.cc:2454] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6213c1dff654d2f9e08d9acce6da51f in term 1.
I20260619 06:38:31.588132 14701 raft_consensus.cc:2454] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6213c1dff654d2f9e08d9acce6da51f in term 1.
I20260619 06:38:31.588352 14765 leader_election.cc:302] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6213c1dff654d2f9e08d9acce6da51f, fa3fa622e01246a2afbf1f2e8cd691ab; no voters:
I20260619 06:38:31.588646 14902 raft_consensus.cc:2781] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:31.589125 14902 raft_consensus.cc:686] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 1 LEADER]: Becoming Leader. State: Replica: c6213c1dff654d2f9e08d9acce6da51f, State: Running, Role: LEADER
I20260619 06:38:31.589586 14902 consensus_queue.cc:237] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 } } peers { permanent_uuid: "c6213c1dff654d2f9e08d9acce6da51f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34467 } } peers { permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 } }
I20260619 06:38:31.594259 14426 catalog_manager.cc:5495] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f reported cstate change: term changed from 0 to 1, leader changed from <none> to c6213c1dff654d2f9e08d9acce6da51f (127.28.104.195). New cstate: current_term: 1 leader_uuid: "c6213c1dff654d2f9e08d9acce6da51f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6213c1dff654d2f9e08d9acce6da51f" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 34467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 } health_report { overall_health: UNKNOWN } } }
I20260619 06:38:31.611811 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.606507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.28.104.195:34467)
0619 06:38:31.606699 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.606705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.606853 (+ 148us) server_negotiation.cc:407] Connection header received
0619 06:38:31.607014 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.607018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.607093 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.607244 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:38:31.607566 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.608325 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.609864 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.610325 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.610438 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.610862 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:38:31.611372 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:38:31.611490 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.611551 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.611616 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 06:38:31.619743 14569 raft_consensus.cc:1261] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab [term 1 FOLLOWER]: Refusing update from remote peer c6213c1dff654d2f9e08d9acce6da51f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:31.619777 14701 raft_consensus.cc:1261] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645 [term 1 FOLLOWER]: Refusing update from remote peer c6213c1dff654d2f9e08d9acce6da51f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:31.620661 14905 consensus_queue.cc:1035] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa3fa622e01246a2afbf1f2e8cd691ab" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:31.621168 14902 consensus_queue.cc:1035] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05dead650d6441a98d0de9b5844bb645" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:31.633607 14909 mvcc.cc:205] Tried to move back new op lower bound from 7298462153184866304 to 7298462153074737152. Current Snapshot: MvccSnapshot[applied={T|T < 7298462153184866304}]
I20260619 06:38:31.645851 14427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:31.646116 14427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:31.649560 14427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca88a50bd11b4d97819445ba3411c821: Sending DeleteTablet for 3 replicas of tablet b6ec03e628d049ffa8cdb90c9c6504e4
I20260619 06:38:31.650609 14681 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ec03e628d049ffa8cdb90c9c6504e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:31 UTC) from {username='slave'} at 127.0.0.1:35426
I20260619 06:38:31.651018 14549 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ec03e628d049ffa8cdb90c9c6504e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:31 UTC) from {username='slave'} at 127.0.0.1:43406
I20260619 06:38:31.651209 14812 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ec03e628d049ffa8cdb90c9c6504e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:31 UTC) from {username='slave'} at 127.0.0.1:45868
I20260619 06:38:31.652325 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 14487
I20260619 06:38:31.663596 14918 tablet_replica.cc:330] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: stopping tablet replica
I20260619 06:38:31.664291 14919 tablet_replica.cc:330] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: stopping tablet replica
I20260619 06:38:31.665618 14918 raft_consensus.cc:2229] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:31.665879 14919 raft_consensus.cc:2229] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:31.666559 14918 raft_consensus.cc:2258] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:31.667137 14919 raft_consensus.cc:2258] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:31.669770 14919 ts_tablet_manager.cc:1905] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:31.670450 14920 tablet_replica.cc:330] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: stopping tablet replica
I20260619 06:38:31.670809 14918 ts_tablet_manager.cc:1905] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:31.671730 14920 raft_consensus.cc:2229] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:38:31.672996 14920 raft_consensus.cc:2258] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:31.678766 14920 ts_tablet_manager.cc:1905] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:31.679283 14918 ts_tablet_manager.cc:1918] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:31.679507 14918 log.cc:1196] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/b6ec03e628d049ffa8cdb90c9c6504e4
I20260619 06:38:31.679558 14919 ts_tablet_manager.cc:1918] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:31.679695 14919 log.cc:1196] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/b6ec03e628d049ffa8cdb90c9c6504e4
I20260619 06:38:31.680231 14918 ts_tablet_manager.cc:1939] T b6ec03e628d049ffa8cdb90c9c6504e4 P 05dead650d6441a98d0de9b5844bb645: Deleting consensus metadata
I20260619 06:38:31.680231 14919 ts_tablet_manager.cc:1939] T b6ec03e628d049ffa8cdb90c9c6504e4 P fa3fa622e01246a2afbf1f2e8cd691ab: Deleting consensus metadata
I20260619 06:38:31.682071 14411 catalog_manager.cc:4841] TS fa3fa622e01246a2afbf1f2e8cd691ab (127.28.104.193:43903): tablet b6ec03e628d049ffa8cdb90c9c6504e4 (table test-table [id=857c3064b25f48ada75d758617ff07e0]) successfully deleted
I20260619 06:38:31.682413 14411 catalog_manager.cc:4841] TS 05dead650d6441a98d0de9b5844bb645 (127.28.104.194:34193): tablet b6ec03e628d049ffa8cdb90c9c6504e4 (table test-table [id=857c3064b25f48ada75d758617ff07e0]) successfully deleted
I20260619 06:38:31.683861 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.653328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.28.104.193:43903)
0619 06:38:31.653499 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.653505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.671981 (+ 18476us) server_negotiation.cc:407] Connection header received
0619 06:38:31.672187 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.672193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.672314 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.672461 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:31.674124 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.675298 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.677302 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.677941 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.678036 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.682271 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:31.682281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:31.682297 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:31.682992 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:31.683038 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:31.683058 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:31.683131 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.683351 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.683455 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:38:31.684554 14609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:31.686172 14920 ts_tablet_manager.cc:1918] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:31.686395 14920 log.cc:1196] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/b6ec03e628d049ffa8cdb90c9c6504e4
I20260619 06:38:31.687220 14920 ts_tablet_manager.cc:1939] T b6ec03e628d049ffa8cdb90c9c6504e4 P c6213c1dff654d2f9e08d9acce6da51f: Deleting consensus metadata
I20260619 06:38:31.689410 14411 catalog_manager.cc:4841] TS c6213c1dff654d2f9e08d9acce6da51f (127.28.104.195:34467): tablet b6ec03e628d049ffa8cdb90c9c6504e4 (table test-table [id=857c3064b25f48ada75d758617ff07e0]) successfully deleted
I20260619 06:38:31.774580 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 14487
I20260619 06:38:31.793467 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 14619
I20260619 06:38:31.801288 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.794429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.28.104.194:34193)
0619 06:38:31.794627 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.794635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.794706 (+ 71us) server_negotiation.cc:407] Connection header received
0619 06:38:31.794846 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.794853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.794971 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.795100 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:31.795548 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.796342 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.797945 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.798366 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.798511 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.799769 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:31.799780 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:31.799794 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:31.800412 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:31.800463 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:31.800484 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:31.800625 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.800789 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.800905 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:38:31.801961 14741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:31.892087 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 14619
I20260619 06:38:31.915743 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 14749
I20260619 06:38:31.923022 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:31.916590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.28.104.195:34467)
0619 06:38:31.916844 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:31.916851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:31.916881 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:38:31.916967 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:31.916974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:31.917126 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:31.917233 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:31.917642 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.918459 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.919771 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:31.920200 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:31.920343 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:31.921563 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:31.921572 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:31.921588 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:31.922197 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:31.922247 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:31.922269 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:31.922414 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:31.922571 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 06:38:31.922667 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 06:38:31.923743 14872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:32.049974 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 14749
I20260619 06:38:32.066131 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 14396
I20260619 06:38:32.067258 14457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:32.178831 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 14396
2026-06-19T06:38:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2695 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:38:32 dist-test-slave-nf4n krb5kdc[14928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:38:32 dist-test-slave-nf4n krb5kdc[14928](info): set up 2 sockets
Jun 19 06:38:32 dist-test-slave-nf4n krb5kdc[14928](info): commencing operation
krb5kdc: starting...
W20260619 06:38:33.300732 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:38:33 dist-test-slave-nf4n krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:38:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:38:33.536952 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:37951
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:37951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:33.855708 14941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:33.855996 14941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:33.856170 14941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:33.863193 14941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:33.863325 14941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:33.863406 14941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:33.863503 14941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:33.863591 14941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:33.873150 14941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:37951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:37951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:33.874423 14941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:33.875710 14941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:33.886739 14949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:33.886829 14947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:33.886957 14946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:33.887812 14941 server_base.cc:1034] running on GCE node
I20260619 06:38:33.888600 14941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:33.889958 14941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:33.891183 14941 hybrid_clock.cc:648] HybridClock initialized: now 1781851113891136 us; error 34 us; skew 500 ppm
I20260619 06:38:33.891518 14941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:33 dist-test-slave-nf4n krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:33.894594 14941 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:38:33.896031 14941 webserver.cc:460] Webserver started at http://127.28.104.254:46177/ using document root <none> and password file <none>
I20260619 06:38:33.896687 14941 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:33.896828 14941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:33.897203 14941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:33.900019 14941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "851dd47859e3477d97a49faea2735c35"
format_stamp: "Formatted at 2026-06-19 06:38:33 on dist-test-slave-nf4n"
server_key: "0c2213248e925707926d13023034d96a"
server_key_iv: "05120b8428256f1f3f6bb0c8d86e78ab"
server_key_version: "encryptionkey@0"
I20260619 06:38:33.900741 14941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "851dd47859e3477d97a49faea2735c35"
format_stamp: "Formatted at 2026-06-19 06:38:33 on dist-test-slave-nf4n"
server_key: "0c2213248e925707926d13023034d96a"
server_key_iv: "05120b8428256f1f3f6bb0c8d86e78ab"
server_key_version: "encryptionkey@0"
I20260619 06:38:33.905620 14941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 06:38:33.908813 14956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:33.910111 14941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:38:33.910290 14941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "851dd47859e3477d97a49faea2735c35"
format_stamp: "Formatted at 2026-06-19 06:38:33 on dist-test-slave-nf4n"
server_key: "0c2213248e925707926d13023034d96a"
server_key_iv: "05120b8428256f1f3f6bb0c8d86e78ab"
server_key_version: "encryptionkey@0"
I20260619 06:38:33.910482 14941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:33.929248 14941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:33.933502 14941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:33.933782 14941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:33.952005 14941 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:37951
I20260619 06:38:33.952013 15009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:37951 every 8 connection(s)
I20260619 06:38:33.953366 14941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:33.957103 15010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:33.958770 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 14941
I20260619 06:38:33.958989 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:33.959627 29091 external_mini_cluster.cc:1428] Setting key 2608390ea4b87d2db84739281a1ef340
I20260619 06:38:33.972807 15010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35: Bootstrap starting.
Jun 19 06:38:33 dist-test-slave-nf4n krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:33.976840 15010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:33.978420 15010 log.cc:824] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:33.981909 15010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35: No bootstrap required, opened a new log
I20260619 06:38:33.984915 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:33.961572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.28.104.254:37951)
0619 06:38:33.962372 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:33.962385 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:33.962429 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:38:33.964052 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:33.964073 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:33.964452 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:33.964776 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:33.966698 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:33.967926 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:33.970042 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:33.970680 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:33.970802 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:33.975939 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:33.975950 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:33.975969 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:33.976031 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:33.980169 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:33.980814 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:33.980824 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:33.980833 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:33.981104 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:33.981478 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:33.981483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:33.981487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:33.982252 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:33.982607 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:33.982982 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 06:38:33.983423 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":378,"threads_started":1}
I20260619 06:38:33.991207 15010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851dd47859e3477d97a49faea2735c35" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37951 } }
I20260619 06:38:33.991545 15010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:33.991623 15010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851dd47859e3477d97a49faea2735c35, State: Initialized, Role: FOLLOWER
I20260619 06:38:33.992116 15010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851dd47859e3477d97a49faea2735c35" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37951 } }
I20260619 06:38:33.992334 15010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:33.992456 15010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:33.992583 15010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:33.994906 15010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851dd47859e3477d97a49faea2735c35" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37951 } }
I20260619 06:38:33.995376 15010 leader_election.cc:302] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 851dd47859e3477d97a49faea2735c35; no voters:
I20260619 06:38:33.995921 15010 leader_election.cc:288] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:33.996097 15015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:33.998232 15015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [term 1 LEADER]: Becoming Leader. State: Replica: 851dd47859e3477d97a49faea2735c35, State: Running, Role: LEADER
I20260619 06:38:33.998579 15010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:33.998879 15015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851dd47859e3477d97a49faea2735c35" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37951 } }
I20260619 06:38:34.002444 15016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "851dd47859e3477d97a49faea2735c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851dd47859e3477d97a49faea2735c35" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37951 } } }
I20260619 06:38:34.002422 15017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 851dd47859e3477d97a49faea2735c35. Latest consensus state: current_term: 1 leader_uuid: "851dd47859e3477d97a49faea2735c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851dd47859e3477d97a49faea2735c35" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 37951 } } }
I20260619 06:38:34.002873 15016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:34.002915 15017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:34.003873 15024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:34.010955 15024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:34.021426 15024 catalog_manager.cc:1347] Generated new cluster ID: f0fc76015acc4c80acecf3fae995d2b2
I20260619 06:38:34.021538 15024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:34.031881 15024 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:34.033213 15024 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:34.045534 15024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35: Generated new TSK 0
I20260619 06:38:34.046418 15024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:38:34.114471 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37951
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:34.447060 15036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:34.447322 15036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:34.447429 15036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:34.454613 15036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:34.454731 15036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:34.454856 15036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:38:34.464305 15036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:34.465596 15036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:34.466823 15036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:34.479641 15044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:34.479679 15042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:34.479610 15041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:34.480677 15036 server_base.cc:1034] running on GCE node
I20260619 06:38:34.481405 15036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:34.489581 15036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:34.490892 15036 hybrid_clock.cc:648] HybridClock initialized: now 1781851114490804 us; error 69 us; skew 500 ppm
I20260619 06:38:34.491272 15036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:34 dist-test-slave-nf4n krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851114, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:34.494829 15036 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:38:34.496286 15036 webserver.cc:460] Webserver started at http://127.28.104.193:40515/ using document root <none> and password file <none>
I20260619 06:38:34.497010 15036 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:34.497236 15036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:34.497565 15036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:34.500577 15036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "776da8159d5447df8fb062365614d254"
format_stamp: "Formatted at 2026-06-19 06:38:34 on dist-test-slave-nf4n"
server_key: "87ba6b820cf037c45b423b12dddf0318"
server_key_iv: "48fb069826a264e23927c4e57f411aa5"
server_key_version: "encryptionkey@0"
I20260619 06:38:34.501380 15036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "776da8159d5447df8fb062365614d254"
format_stamp: "Formatted at 2026-06-19 06:38:34 on dist-test-slave-nf4n"
server_key: "87ba6b820cf037c45b423b12dddf0318"
server_key_iv: "48fb069826a264e23927c4e57f411aa5"
server_key_version: "encryptionkey@0"
I20260619 06:38:34.506668 15036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:38:34.510005 15051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:34.511229 15036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:38:34.511407 15036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "776da8159d5447df8fb062365614d254"
format_stamp: "Formatted at 2026-06-19 06:38:34 on dist-test-slave-nf4n"
server_key: "87ba6b820cf037c45b423b12dddf0318"
server_key_iv: "48fb069826a264e23927c4e57f411aa5"
server_key_version: "encryptionkey@0"
I20260619 06:38:34.511603 15036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:34.532572 15036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:34.536387 15036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:34.536681 15036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:34.537664 15036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:34.538967 15036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:34.539079 15036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:34.539220 15036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:34.539291 15036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:34.571422 15036 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:33607
I20260619 06:38:34.571472 15164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:33607 every 8 connection(s)
I20260619 06:38:34.572769 15036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:38:34.582072 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 15036
I20260619 06:38:34.582250 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:38:34.582700 29091 external_mini_cluster.cc:1428] Setting key ad9041a826da1dee71681138f7f52932
Jun 19 06:38:34 dist-test-slave-nf4n krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851114, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:34.592329 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:34.575368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:50615 (local address 127.28.104.254:37951)
0619 06:38:34.575834 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:34.575839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:34.576689 (+ 850us) server_negotiation.cc:407] Connection header received
0619 06:38:34.578558 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:34.578564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:34.578653 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:34.578781 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:34.580305 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:34.581105 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:34.582731 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:34.583188 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:34.583328 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:34.586752 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:34.586757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:34.586761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:34.586790 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:34.589373 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:34.590076 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:34.590083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:34.590087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:34.590206 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:34.590632 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:34.590639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:34.590642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:34.590901 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:34.591057 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:34.591730 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 06:38:34.591949 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":239,"threads_started":1}
I20260619 06:38:34.593318 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:34.575841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37951 (local address 127.28.104.193:50615)
0619 06:38:34.576529 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:34.576566 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:34.578285 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:34.579013 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:34.579021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:34.579572 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:34.580131 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:34.580148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:34.581274 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:34.581278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:34.582592 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:34.582605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:34.583416 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:34.583426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:34.583660 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:34.584388 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:38:34.584418 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:34.586544 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:34.589505 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:34.589514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:34.589530 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:34.589958 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:34.590327 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:34.590332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:34.590336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:34.590522 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:34.591031 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:34.591038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:34.591375 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 06:38:34.591632 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 06:38:34.592149 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":256,"threads_started":1}
I20260619 06:38:34.594830 15165 heartbeater.cc:347] Connected to a master server at 127.28.104.254:37951
I20260619 06:38:34.595118 15165 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:34.595798 15165 heartbeater.cc:511] Master 127.28.104.254:37951 requested a full tablet report, sending...
I20260619 06:38:34.597848 14973 ts_manager.cc:194] Registered new tserver with Master: 776da8159d5447df8fb062365614d254 (127.28.104.193:33607)
I20260619 06:38:34.599433 14973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:50615
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:38:34.633657 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37951
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:34.960438 15171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:34.960758 15171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:34.961162 15171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:34.968797 15171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:34.968964 15171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:34.969187 15171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:38:34.979254 15171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:34.980600 15171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:34.982107 15171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:34.995465 15178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:34.995545 15180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:34.995496 15177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:34.996745 15171 server_base.cc:1034] running on GCE node
I20260619 06:38:34.997485 15171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:34.998492 15171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:34.999734 15171 hybrid_clock.cc:648] HybridClock initialized: now 1781851114999637 us; error 80 us; skew 500 ppm
I20260619 06:38:35.000104 15171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:35 dist-test-slave-nf4n krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851115, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:35.007823 15171 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:38:35.009598 15171 webserver.cc:460] Webserver started at http://127.28.104.194:33161/ using document root <none> and password file <none>
I20260619 06:38:35.010295 15171 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:35.010424 15171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:35.010764 15171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:35.013839 15171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "f3de30391baf41748efdfd241a72dc23"
format_stamp: "Formatted at 2026-06-19 06:38:35 on dist-test-slave-nf4n"
server_key: "e4444c6ed60dcfc53bddf55bd5692b95"
server_key_iv: "bf980170013c13b5c12e4057a8516531"
server_key_version: "encryptionkey@0"
I20260619 06:38:35.014635 15171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "f3de30391baf41748efdfd241a72dc23"
format_stamp: "Formatted at 2026-06-19 06:38:35 on dist-test-slave-nf4n"
server_key: "e4444c6ed60dcfc53bddf55bd5692b95"
server_key_iv: "bf980170013c13b5c12e4057a8516531"
server_key_version: "encryptionkey@0"
I20260619 06:38:35.020123 15171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:38:35.023497 15187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:35.024801 15171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:38:35.024955 15171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "f3de30391baf41748efdfd241a72dc23"
format_stamp: "Formatted at 2026-06-19 06:38:35 on dist-test-slave-nf4n"
server_key: "e4444c6ed60dcfc53bddf55bd5692b95"
server_key_iv: "bf980170013c13b5c12e4057a8516531"
server_key_version: "encryptionkey@0"
I20260619 06:38:35.025216 15171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:35.037204 15171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:35.041144 15171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:35.041425 15171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:35.042337 15171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:35.043586 15171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:35.043660 15171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:35.043825 15171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:35.043889 15171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:35.076771 15171 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:46373
I20260619 06:38:35.076874 15299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:46373 every 8 connection(s)
I20260619 06:38:35.078074 15171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:38:35.086328 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 15171
I20260619 06:38:35.086520 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:38:35.086985 29091 external_mini_cluster.cc:1428] Setting key ce6e6644fc27e5ef11f7df71ff4301bf
Jun 19 06:38:35 dist-test-slave-nf4n krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851115, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:35.099493 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.080814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:40701 (local address 127.28.104.254:37951)
0619 06:38:35.081130 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:35.081134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:35.082369 (+ 1235us) server_negotiation.cc:407] Connection header received
0619 06:38:35.084195 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:35.084200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:35.084299 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:35.084407 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:35.086019 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.086843 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.088524 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.089106 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.089237 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.092577 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:35.092586 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:35.092594 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:35.092637 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:35.096399 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:35.097132 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:35.097137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:35.097140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:35.097279 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:35.097738 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:35.097741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:35.097743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:35.097992 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:35.098166 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:35.098865 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 06:38:35.099147 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260619 06:38:35.100446 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.081473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37951 (local address 127.28.104.194:40701)
0619 06:38:35.082197 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:35.082235 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:35.083927 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:35.084635 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:35.084642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:35.085266 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:35.085829 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.085846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.086985 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.086991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.088386 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.088398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.089314 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.089319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.089518 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.090235 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:38:35.090257 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:35.092340 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:35.096535 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:35.096541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:35.096554 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:35.096958 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:35.097396 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:35.097401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:35.097405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:35.097585 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:35.098118 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:35.098125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:35.098488 (+ 363us) client_negotiation.cc:769] Sending connection context
0619 06:38:35.098755 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 06:38:35.099278 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":305,"threads_started":1}
I20260619 06:38:35.101956 15300 heartbeater.cc:347] Connected to a master server at 127.28.104.254:37951
I20260619 06:38:35.102275 15300 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:35.103127 15300 heartbeater.cc:511] Master 127.28.104.254:37951 requested a full tablet report, sending...
I20260619 06:38:35.104702 14973 ts_manager.cc:194] Registered new tserver with Master: f3de30391baf41748efdfd241a72dc23 (127.28.104.194:46373)
I20260619 06:38:35.105670 14973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:40701
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:38:35.143792 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37951
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:35.469617 15305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:35.469890 15305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:35.469987 15305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:35.477272 15305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:35.477371 15305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:35.477487 15305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:38:35.486888 15305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:37951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:35.488037 15305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:35.489388 15305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:35.502615 15311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:35.502759 15310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:35.502734 15313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:35.503608 15305 server_base.cc:1034] running on GCE node
I20260619 06:38:35.504195 15305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:35.505242 15305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:35.506448 15305 hybrid_clock.cc:648] HybridClock initialized: now 1781851115506387 us; error 55 us; skew 500 ppm
I20260619 06:38:35.506796 15305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:35 dist-test-slave-nf4n krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851115, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:35.509892 15305 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:38:35.511241 15305 webserver.cc:460] Webserver started at http://127.28.104.195:44697/ using document root <none> and password file <none>
I20260619 06:38:35.511868 15305 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:35.511965 15305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:35.512214 15305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:35.515305 15305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "b4ca8090e5b2423186c2cb27dbe3cee1"
format_stamp: "Formatted at 2026-06-19 06:38:35 on dist-test-slave-nf4n"
server_key: "458b5b063a6722533db7ae1eba1a648b"
server_key_iv: "97d3a2b5ab3dadf3e5aa3f0ee7c32533"
server_key_version: "encryptionkey@0"
I20260619 06:38:35.516006 15305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "b4ca8090e5b2423186c2cb27dbe3cee1"
format_stamp: "Formatted at 2026-06-19 06:38:35 on dist-test-slave-nf4n"
server_key: "458b5b063a6722533db7ae1eba1a648b"
server_key_iv: "97d3a2b5ab3dadf3e5aa3f0ee7c32533"
server_key_version: "encryptionkey@0"
I20260619 06:38:35.520991 15305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:38:35.524349 15320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:35.525621 15305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:38:35.525779 15305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "b4ca8090e5b2423186c2cb27dbe3cee1"
format_stamp: "Formatted at 2026-06-19 06:38:35 on dist-test-slave-nf4n"
server_key: "458b5b063a6722533db7ae1eba1a648b"
server_key_iv: "97d3a2b5ab3dadf3e5aa3f0ee7c32533"
server_key_version: "encryptionkey@0"
I20260619 06:38:35.525976 15305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:35.547434 15305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:35.551760 15305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:35.552050 15305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:35.553013 15305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:35.554340 15305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:35.554437 15305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:35.554576 15305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:35.554646 15305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:35.587111 15305 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:38017
I20260619 06:38:35.587165 15433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:38017 every 8 connection(s)
I20260619 06:38:35.588404 15305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:38:35.598028 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 15305
I20260619 06:38:35.598240 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:38:35.598798 29091 external_mini_cluster.cc:1428] Setting key 6fa1712c104d0879179d843490304ea1
Jun 19 06:38:35 dist-test-slave-nf4n krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851115, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:35.602309 15165 heartbeater.cc:503] Master 127.28.104.254:37951 was elected leader, sending a full tablet report...
I20260619 06:38:35.608376 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.591055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:45891 (local address 127.28.104.254:37951)
0619 06:38:35.591232 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:35.591237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:35.592331 (+ 1094us) server_negotiation.cc:407] Connection header received
0619 06:38:35.594232 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:35.594240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:35.594337 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:35.594451 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:35.595926 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.596681 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.598291 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.598991 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.599128 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.602940 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:35.602949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:35.602954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:35.602995 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:35.605697 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:35.606471 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:35.606477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:35.606482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:35.606611 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:35.606978 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:35.606981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:35.606983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:35.607228 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:35.607338 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:35.607979 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 06:38:35.608134 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:38:35.609438 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.591486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:37951 (local address 127.28.104.195:45891)
0619 06:38:35.592161 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:35.592206 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:35.593962 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:35.594681 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:35.594689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:35.595204 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:35.595752 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.595769 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.596835 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.596842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.598114 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.598127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.599197 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.599201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.599533 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.600310 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:38:35.600332 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:35.602720 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:35.605831 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:35.605838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:35.605850 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:35.606343 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:35.606702 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:35.606706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:35.606708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:35.606881 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:35.607336 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:35.607341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:35.607636 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 06:38:35.607883 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 06:38:35.608239 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":257,"threads_started":1}
I20260619 06:38:35.610723 15434 heartbeater.cc:347] Connected to a master server at 127.28.104.254:37951
I20260619 06:38:35.610978 15434 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:35.611609 15434 heartbeater.cc:511] Master 127.28.104.254:37951 requested a full tablet report, sending...
I20260619 06:38:35.613216 14973 ts_manager.cc:194] Registered new tserver with Master: b4ca8090e5b2423186c2cb27dbe3cee1 (127.28.104.195:38017)
I20260619 06:38:35.614128 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:38:35.614279 14973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:45891
I20260619 06:38:35.631282 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.618489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.28.104.254:37951)
0619 06:38:35.618646 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:35.618650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:35.619188 (+ 538us) server_negotiation.cc:407] Connection header received
0619 06:38:35.619319 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:35.619322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:35.619377 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:35.619453 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:35.620797 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.621596 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.622869 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.623276 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.623376 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.625936 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:35.625941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:35.625944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:35.625974 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:35.628552 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:35.629372 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:35.629376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:35.629378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:35.629554 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:35.629968 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:35.629971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:35.629973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:35.630380 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:35.630612 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:35.630900 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 06:38:35.631029 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:38:35.634974 14973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:38:35.641779 14973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:38:35.681072 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.653779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:38017 (local address 127.0.0.1:36102)
0619 06:38:35.670921 (+ 17142us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:35.670951 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:35.671150 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:35.674238 (+ 3088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:35.674242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:35.674261 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:35.674661 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.674674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.676574 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.676579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.678767 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.678777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.679991 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.679997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.680173 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.680229 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 06:38:35.680367 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 06:38:35.680573 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16969,"mutex_wait_us":168,"thread_start_us":18899,"threads_started":1}
I20260619 06:38:35.681257 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.654149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.28.104.194:46373)
0619 06:38:35.672146 (+ 17997us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:35.672160 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:35.672197 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:38:35.672823 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:35.672836 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:35.673214 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:35.673433 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:35.674104 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.675694 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.678662 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.680055 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.680194 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.680335 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:35.680716 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 06:38:35.680903 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17770,"thread_start_us":18063,"threads_started":1}
I20260619 06:38:35.681562 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.654116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.28.104.195:38017)
0619 06:38:35.673511 (+ 19395us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:35.673521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:35.673549 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:38:35.673625 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:35.673631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:35.673814 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:35.673972 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:35.674841 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.676410 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.678911 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.679800 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.680012 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.680188 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:35.681044 (+ 856us) server_negotiation.cc:299] Negotiation successful
0619 06:38:35.681235 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19218,"thread_start_us":19066,"threads_started":1}
I20260619 06:38:35.681701 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.655097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.28.104.193:33607)
0619 06:38:35.669578 (+ 14481us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:35.669587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:35.673425 (+ 3838us) server_negotiation.cc:407] Connection header received
0619 06:38:35.673534 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:35.673542 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:35.673752 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:35.673950 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:35.674660 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.675714 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.678445 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.679950 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.680148 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.680336 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:35.681171 (+ 835us) server_negotiation.cc:299] Negotiation successful
0619 06:38:35.681368 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14263,"thread_start_us":15047,"threads_started":1}
I20260619 06:38:35.682139 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.653779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:46373 (local address 127.0.0.1:42368)
0619 06:38:35.668949 (+ 15170us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:35.669016 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:35.672517 (+ 3501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:35.673486 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:35.673496 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:35.673530 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:35.673893 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.673907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.675858 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.675864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.678503 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.678517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.680356 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.680363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.680527 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.680562 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:38:35.680644 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:38:35.680776 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14825,"mutex_wait_us":1,"thread_start_us":16786,"threads_started":1}
I20260619 06:38:35.682175 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.654340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:33607 (local address 127.0.0.1:57106)
0619 06:38:35.673139 (+ 18799us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:35.673163 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:35.673282 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:35.673950 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:35.673955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:35.673984 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:35.674428 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.674442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.675888 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.675894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.678297 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.678308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.680481 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.680486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.680645 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.680680 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:38:35.680770 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 06:38:35.680899 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18635,"mutex_wait_us":130,"thread_start_us":14111,"threads_started":1}
I20260619 06:38:35.685509 15235 tablet_service.cc:1467] Processing CreateTablet for tablet e47afd73eb2241d3a603b3b018aa7bba (DEFAULT_TABLE table=test-table [id=db4c6224eeeb40d395a39a43fccd3708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:35.685539 15369 tablet_service.cc:1467] Processing CreateTablet for tablet e47afd73eb2241d3a603b3b018aa7bba (DEFAULT_TABLE table=test-table [id=db4c6224eeeb40d395a39a43fccd3708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:35.686729 15235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47afd73eb2241d3a603b3b018aa7bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:35.686965 15369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47afd73eb2241d3a603b3b018aa7bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:35.686546 15100 tablet_service.cc:1467] Processing CreateTablet for tablet e47afd73eb2241d3a603b3b018aa7bba (DEFAULT_TABLE table=test-table [id=db4c6224eeeb40d395a39a43fccd3708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:35.687759 15100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47afd73eb2241d3a603b3b018aa7bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:35.697691 15452 tablet_bootstrap.cc:492] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: Bootstrap starting.
I20260619 06:38:35.701012 15452 tablet_bootstrap.cc:654] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:35.701177 15453 tablet_bootstrap.cc:492] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: Bootstrap starting.
I20260619 06:38:35.702392 15452 log.cc:824] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:35.702760 15454 tablet_bootstrap.cc:492] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: Bootstrap starting.
I20260619 06:38:35.704960 15453 tablet_bootstrap.cc:654] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:35.705955 15454 tablet_bootstrap.cc:654] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:35.706204 15453 log.cc:824] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:35.706403 15452 tablet_bootstrap.cc:492] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: No bootstrap required, opened a new log
I20260619 06:38:35.706658 15452 ts_tablet_manager.cc:1397] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260619 06:38:35.707161 15454 log.cc:824] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:35.709193 15453 tablet_bootstrap.cc:492] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: No bootstrap required, opened a new log
I20260619 06:38:35.709493 15453 ts_tablet_manager.cc:1397] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 06:38:35.710187 15454 tablet_bootstrap.cc:492] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: No bootstrap required, opened a new log
I20260619 06:38:35.710436 15454 ts_tablet_manager.cc:1397] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 06:38:35.717396 15452 raft_consensus.cc:348] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 } } peers { permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 } } peers { permanent_uuid: "776da8159d5447df8fb062365614d254" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 33607 } }
I20260619 06:38:35.717872 15452 raft_consensus.cc:374] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:35.717967 15452 raft_consensus.cc:729] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3de30391baf41748efdfd241a72dc23, State: Initialized, Role: FOLLOWER
I20260619 06:38:35.718545 15454 raft_consensus.cc:348] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 } } peers { permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 } } peers { permanent_uuid: "776da8159d5447df8fb062365614d254" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 33607 } }
I20260619 06:38:35.718598 15452 consensus_queue.cc:260] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 } } peers { permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 } } peers { permanent_uuid: "776da8159d5447df8fb062365614d254" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 33607 } }
I20260619 06:38:35.718863 15454 raft_consensus.cc:374] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:35.718990 15454 raft_consensus.cc:729] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 776da8159d5447df8fb062365614d254, State: Initialized, Role: FOLLOWER
I20260619 06:38:35.719564 15454 consensus_queue.cc:260] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 } } peers { permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 } } peers { permanent_uuid: "776da8159d5447df8fb062365614d254" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 33607 } }
I20260619 06:38:35.720187 15453 raft_consensus.cc:348] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 } } peers { permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 } } peers { permanent_uuid: "776da8159d5447df8fb062365614d254" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 33607 } }
I20260619 06:38:35.720736 15453 raft_consensus.cc:374] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:35.720863 15453 raft_consensus.cc:729] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ca8090e5b2423186c2cb27dbe3cee1, State: Initialized, Role: FOLLOWER
I20260619 06:38:35.721370 15454 ts_tablet_manager.cc:1428] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 06:38:35.721705 15453 consensus_queue.cc:260] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 } } peers { permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 } } peers { permanent_uuid: "776da8159d5447df8fb062365614d254" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 33607 } }
I20260619 06:38:35.725857 15300 heartbeater.cc:503] Master 127.28.104.254:37951 was elected leader, sending a full tablet report...
I20260619 06:38:35.727046 15452 ts_tablet_manager.cc:1428] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260619 06:38:35.728056 15434 heartbeater.cc:503] Master 127.28.104.254:37951 was elected leader, sending a full tablet report...
I20260619 06:38:35.728614 15453 ts_tablet_manager.cc:1428] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
W20260619 06:38:35.825310 15166 tablet.cc:2369] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:38:35.830164 15301 tablet.cc:2369] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:38:35.840015 15435 tablet.cc:2369] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:38:35.935530 15459 raft_consensus.cc:482] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:38:35.935819 15459 raft_consensus.cc:504] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 } } peers { permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 } } peers { permanent_uuid: "776da8159d5447df8fb062365614d254" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 33607 } }
I20260619 06:38:35.937628 15459 leader_election.cc:288] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4ca8090e5b2423186c2cb27dbe3cee1 (127.28.104.195:38017), f3de30391baf41748efdfd241a72dc23 (127.28.104.194:46373)
I20260619 06:38:35.946906 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.938213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:41395 (local address 127.28.104.194:46373)
0619 06:38:35.938442 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:35.938451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:35.938953 (+ 502us) server_negotiation.cc:407] Connection header received
0619 06:38:35.939212 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:35.939217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:35.939284 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:35.939368 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:35.940202 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.941323 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.944473 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.945766 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.945923 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.946000 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:35.946505 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 06:38:35.946637 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:38:35.946906 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.938018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:48169 (local address 127.28.104.195:38017)
0619 06:38:35.938233 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:35.938240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:35.939043 (+ 803us) server_negotiation.cc:407] Connection header received
0619 06:38:35.939191 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:35.939196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:35.939273 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:35.939377 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:35.940139 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.941331 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.944496 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.945766 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.945925 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.946000 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:35.946505 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 06:38:35.946637 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:38:35.947168 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.937865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:46373 (local address 127.28.104.193:41395)
0619 06:38:35.938841 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:35.938863 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:35.939024 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:35.939579 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:35.939585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:35.939622 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:35.940056 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.940066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.941462 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.941466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.944243 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.944259 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.946058 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.946063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.946304 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.946359 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 06:38:35.946458 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:38:35.946643 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":235,"thread_start_us":640,"threads_started":1}
I20260619 06:38:35.947192 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:35.937865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:38017 (local address 127.28.104.193:48169)
0619 06:38:35.938933 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:35.938947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:35.939056 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:35.939429 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:35.939435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:35.939461 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:35.939921 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.939935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.941460 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.941465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.944250 (+ 2785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:35.944259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:35.946043 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:35.946050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:35.946310 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:35.946364 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 06:38:35.946458 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:38:35.946643 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":382,"spinlock_wait_cycles":768,"thread_start_us":550,"threads_started":1}
I20260619 06:38:35.948243 15389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47afd73eb2241d3a603b3b018aa7bba" candidate_uuid: "776da8159d5447df8fb062365614d254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" is_pre_election: true
I20260619 06:38:35.948240 15255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47afd73eb2241d3a603b3b018aa7bba" candidate_uuid: "776da8159d5447df8fb062365614d254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3de30391baf41748efdfd241a72dc23" is_pre_election: true
I20260619 06:38:35.948722 15389 raft_consensus.cc:2454] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 776da8159d5447df8fb062365614d254 in term 0.
I20260619 06:38:35.948724 15255 raft_consensus.cc:2454] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 776da8159d5447df8fb062365614d254 in term 0.
I20260619 06:38:35.949455 15054 leader_election.cc:302] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 776da8159d5447df8fb062365614d254, b4ca8090e5b2423186c2cb27dbe3cee1, f3de30391baf41748efdfd241a72dc23; no voters:
I20260619 06:38:35.949859 15459 raft_consensus.cc:2781] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:38:35.950004 15459 raft_consensus.cc:482] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:38:35.950119 15459 raft_consensus.cc:3037] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:35.952708 15459 raft_consensus.cc:504] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 } } peers { permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 } } peers { permanent_uuid: "776da8159d5447df8fb062365614d254" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 33607 } }
I20260619 06:38:35.953423 15459 leader_election.cc:288] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [CANDIDATE]: Term 1 election: Requested vote from peers b4ca8090e5b2423186c2cb27dbe3cee1 (127.28.104.195:38017), f3de30391baf41748efdfd241a72dc23 (127.28.104.194:46373)
I20260619 06:38:35.953855 15389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47afd73eb2241d3a603b3b018aa7bba" candidate_uuid: "776da8159d5447df8fb062365614d254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1"
I20260619 06:38:35.953891 15255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47afd73eb2241d3a603b3b018aa7bba" candidate_uuid: "776da8159d5447df8fb062365614d254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3de30391baf41748efdfd241a72dc23"
I20260619 06:38:35.954073 15255 raft_consensus.cc:3037] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:35.954071 15389 raft_consensus.cc:3037] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:35.957257 15255 raft_consensus.cc:2454] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 776da8159d5447df8fb062365614d254 in term 1.
I20260619 06:38:35.957257 15389 raft_consensus.cc:2454] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 776da8159d5447df8fb062365614d254 in term 1.
I20260619 06:38:35.957641 15054 leader_election.cc:302] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 776da8159d5447df8fb062365614d254, f3de30391baf41748efdfd241a72dc23; no voters:
I20260619 06:38:35.958027 15459 raft_consensus.cc:2781] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:35.958540 15459 raft_consensus.cc:686] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 1 LEADER]: Becoming Leader. State: Replica: 776da8159d5447df8fb062365614d254, State: Running, Role: LEADER
I20260619 06:38:35.959036 15459 consensus_queue.cc:237] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 } } peers { permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 } } peers { permanent_uuid: "776da8159d5447df8fb062365614d254" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 33607 } }
I20260619 06:38:35.963775 14972 catalog_manager.cc:5495] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 reported cstate change: term changed from 0 to 1, leader changed from <none> to 776da8159d5447df8fb062365614d254 (127.28.104.193). New cstate: current_term: 1 leader_uuid: "776da8159d5447df8fb062365614d254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "776da8159d5447df8fb062365614d254" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 33607 } health_report { overall_health: HEALTHY } } }
I20260619 06:38:36.035097 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.029798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.28.104.193:33607)
0619 06:38:36.029971 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:36.029976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:36.030055 (+ 79us) server_negotiation.cc:407] Connection header received
0619 06:38:36.030219 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:36.030222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:36.030303 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:36.030427 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:38:36.030772 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:36.031492 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:36.032764 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:36.033321 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:36.033440 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:36.034025 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:38:36.034518 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:38:36.034702 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:36.034828 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 06:38:36.034875 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:38:36.043926 15255 raft_consensus.cc:1261] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23 [term 1 FOLLOWER]: Refusing update from remote peer 776da8159d5447df8fb062365614d254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:36.043927 15389 raft_consensus.cc:1261] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1 [term 1 FOLLOWER]: Refusing update from remote peer 776da8159d5447df8fb062365614d254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:36.044919 15463 consensus_queue.cc:1035] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4ca8090e5b2423186c2cb27dbe3cee1" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:36.045327 15459 consensus_queue.cc:1035] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3de30391baf41748efdfd241a72dc23" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:36.056018 15466 mvcc.cc:205] Tried to move back new op lower bound from 7298462171305021440 to 7298462170972024832. Current Snapshot: MvccSnapshot[applied={T|T < 7298462171305021440}]
I20260619 06:38:36.069934 14972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:36.070194 14972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:36.073923 14972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 851dd47859e3477d97a49faea2735c35: Sending DeleteTablet for 3 replicas of tablet e47afd73eb2241d3a603b3b018aa7bba
I20260619 06:38:36.075058 15369 tablet_service.cc:1514] Processing DeleteTablet for tablet e47afd73eb2241d3a603b3b018aa7bba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:36 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:36102
I20260619 06:38:36.075114 15235 tablet_service.cc:1514] Processing DeleteTablet for tablet e47afd73eb2241d3a603b3b018aa7bba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:36 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:42368
I20260619 06:38:36.075312 15100 tablet_service.cc:1514] Processing DeleteTablet for tablet e47afd73eb2241d3a603b3b018aa7bba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:36 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:57106
I20260619 06:38:36.080561 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 15036
I20260619 06:38:36.103114 15478 tablet_replica.cc:330] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: stopping tablet replica
I20260619 06:38:36.103969 15477 tablet_replica.cc:330] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: stopping tablet replica
I20260619 06:38:36.104693 15479 tablet_replica.cc:330] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: stopping tablet replica
I20260619 06:38:36.104928 15478 raft_consensus.cc:2229] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:36.105512 15478 raft_consensus.cc:2258] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:36.107765 15478 ts_tablet_manager.cc:1905] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:36.105758 15479 raft_consensus.cc:2229] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:38:36.109349 15479 raft_consensus.cc:2258] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:36.109709 15477 raft_consensus.cc:2229] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:36.110231 15477 raft_consensus.cc:2258] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 06:38:36 dist-test-slave-nf4n krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:38:36.115609 15477 ts_tablet_manager.cc:1905] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:36.120172 15479 ts_tablet_manager.cc:1905] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:36.120713 15478 ts_tablet_manager.cc:1918] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:36.121611 15478 log.cc:1196] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/e47afd73eb2241d3a603b3b018aa7bba
I20260619 06:38:36.122218 15478 ts_tablet_manager.cc:1939] T e47afd73eb2241d3a603b3b018aa7bba P f3de30391baf41748efdfd241a72dc23: Deleting consensus metadata
I20260619 06:38:36.124102 14959 catalog_manager.cc:4841] TS f3de30391baf41748efdfd241a72dc23 (127.28.104.194:46373): tablet e47afd73eb2241d3a603b3b018aa7bba (table test-table [id=db4c6224eeeb40d395a39a43fccd3708]) successfully deleted
I20260619 06:38:36.125212 15477 ts_tablet_manager.cc:1918] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:36.125445 15477 log.cc:1196] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/e47afd73eb2241d3a603b3b018aa7bba
I20260619 06:38:36.125923 15477 ts_tablet_manager.cc:1939] T e47afd73eb2241d3a603b3b018aa7bba P b4ca8090e5b2423186c2cb27dbe3cee1: Deleting consensus metadata
I20260619 06:38:36.127295 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.084404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.28.104.193:33607)
0619 06:38:36.084627 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:36.084636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:36.099908 (+ 15272us) server_negotiation.cc:407] Connection header received
0619 06:38:36.100443 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:36.100453 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:36.100573 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:36.100666 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:36.103725 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:36.105062 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:36.106996 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:36.107725 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:36.107828 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:36.115182 (+ 7354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:36.115195 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:36.115211 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:36.115263 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:36.119213 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:36.124554 (+ 5341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:36.124563 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:36.124570 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:36.124771 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:36.125375 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:36.125383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:36.125387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:36.125923 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:36.126246 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:36.126642 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 06:38:36.126930 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:38:36.127779 14960 catalog_manager.cc:4841] TS b4ca8090e5b2423186c2cb27dbe3cee1 (127.28.104.195:38017): tablet e47afd73eb2241d3a603b3b018aa7bba (table test-table [id=db4c6224eeeb40d395a39a43fccd3708]) successfully deleted
I20260619 06:38:36.128546 15479 ts_tablet_manager.cc:1918] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:36.128767 15479 log.cc:1196] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/e47afd73eb2241d3a603b3b018aa7bba
I20260619 06:38:36.128944 15160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:36.226881 15479 ts_tablet_manager.cc:1939] T e47afd73eb2241d3a603b3b018aa7bba P 776da8159d5447df8fb062365614d254: Deleting consensus metadata
I20260619 06:38:36.227464 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 15036
W20260619 06:38:36.245417 14958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.104.193:33607 (error 108)
I20260619 06:38:36.245627 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 15171
I20260619 06:38:36.247654 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.246964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:33607 (local address 127.0.0.1:57134)
0619 06:38:36.247393 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:36.247500 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:33607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":237,"threads_started":1}
W20260619 06:38:36.247867 14958 catalog_manager.cc:4584] TS 776da8159d5447df8fb062365614d254 (127.28.104.193:33607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e47afd73eb2241d3a603b3b018aa7bba: Network error: Client connection negotiation failed: client connection to 127.28.104.193:33607: connect: Connection refused (error 111)
Jun 19 06:38:36 dist-test-slave-nf4n krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:38:36.260574 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.246396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.28.104.194:46373)
0619 06:38:36.246578 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:36.246582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:36.246675 (+ 93us) server_negotiation.cc:407] Connection header received
0619 06:38:36.246819 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:36.246823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:36.246884 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:36.247010 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:36.248445 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:36.249190 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:36.250498 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:36.250895 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:36.251029 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:36.255232 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:36.255259 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:36.255274 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:36.255306 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:36.257877 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:36.258538 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:36.258545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:36.258548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:36.258676 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:36.259045 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:36.259051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:36.259054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:36.259416 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:36.259724 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:36.260017 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 06:38:36.260231 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:38:36.261364 15295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:36.301247 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.300771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:33607 (local address 127.0.0.1:57138)
0619 06:38:36.300973 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:36.301108 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:33607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 06:38:36.302228 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.301897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:33607 (local address 127.0.0.1:57150)
0619 06:38:36.302037 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:36.302102 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:33607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 06:38:36.351776 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 15171
I20260619 06:38:36.362929 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.362336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:33607 (local address 127.0.0.1:57164)
0619 06:38:36.362593 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:36.362718 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:33607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 06:38:36.364106 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.363618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:33607 (local address 127.0.0.1:57174)
0619 06:38:36.363809 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:36.363907 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:33607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 06:38:36.369699 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 15305
Jun 19 06:38:36 dist-test-slave-nf4n krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:38:36.384511 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.370485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.28.104.195:38017)
0619 06:38:36.370679 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:36.370684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:36.370706 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:38:36.370900 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:36.370904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:36.370977 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:36.371084 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:36.372443 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:36.373237 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:36.374509 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:36.374912 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:36.375041 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:36.379466 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:36.379482 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:36.379498 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:36.379532 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:36.382171 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:36.382825 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:36.382832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:36.382834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:36.382963 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:36.383279 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:36.383282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:36.383284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:36.383639 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:36.383851 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:36.384092 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 06:38:36.384267 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:38:36.385237 15429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:36.444700 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.444170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:33607 (local address 127.0.0.1:57188)
0619 06:38:36.444411 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:36.444529 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:33607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 06:38:36.445901 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:36.445438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:33607 (local address 127.0.0.1:57198)
0619 06:38:36.445644 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:36.445733 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:33607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 06:38:36.473472 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 15305
I20260619 06:38:36.490103 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 14941
I20260619 06:38:36.491264 15004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:36.603401 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 14941
2026-06-19T06:38:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4426 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T06:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:36Z Disabled control of system clock
I20260619 06:38:36.678519 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:33215
--webserver_interface=127.28.104.254
--webserver_port=0
--builtin_ntp_servers=127.28.104.212:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:33215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:38:37.009510 15490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:37.009799 15490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:37.009948 15490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:37.017416 15490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:37.017525 15490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:37.017607 15490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:37.017715 15490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:37.017798 15490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:37.027001 15490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:33215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:33215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:37.028352 15490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:37.030109 15490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:37.041617 15495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:37.041687 15498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:37.041757 15496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:37.042774 15490 server_base.cc:1034] running on GCE node
I20260619 06:38:37.043476 15490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:37.044744 15490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:37.045958 15490 hybrid_clock.cc:648] HybridClock initialized: now 1781851117045896 us; error 59 us; skew 500 ppm
I20260619 06:38:37.046275 15490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:37.051327 15490 webserver.cc:460] Webserver started at http://127.28.104.254:43349/ using document root <none> and password file <none>
I20260619 06:38:37.051980 15490 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:37.052065 15490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:37.052315 15490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:37.055110 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "7d52e37e63dd408f95007aa3dcfa5b2d"
format_stamp: "Formatted at 2026-06-19 06:38:37 on dist-test-slave-nf4n"
server_key: "11718450aeb56a660b57dbb760f80df4"
server_key_iv: "2bdc05c74af1ab9f319ab13c780694a5"
server_key_version: "encryptionkey@0"
I20260619 06:38:37.055837 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "7d52e37e63dd408f95007aa3dcfa5b2d"
format_stamp: "Formatted at 2026-06-19 06:38:37 on dist-test-slave-nf4n"
server_key: "11718450aeb56a660b57dbb760f80df4"
server_key_iv: "2bdc05c74af1ab9f319ab13c780694a5"
server_key_version: "encryptionkey@0"
I20260619 06:38:37.060602 15490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 06:38:37.063892 15504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:37.065305 15490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:38:37.065486 15490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "7d52e37e63dd408f95007aa3dcfa5b2d"
format_stamp: "Formatted at 2026-06-19 06:38:37 on dist-test-slave-nf4n"
server_key: "11718450aeb56a660b57dbb760f80df4"
server_key_iv: "2bdc05c74af1ab9f319ab13c780694a5"
server_key_version: "encryptionkey@0"
I20260619 06:38:37.065691 15490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:37.091015 15490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:37.091871 15490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:37.092159 15490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:37.116698 15490 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:33215
I20260619 06:38:37.116732 15555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:33215 every 8 connection(s)
I20260619 06:38:37.118666 15490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:37.119926 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 15490
I20260619 06:38:37.120093 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:37.120548 29091 external_mini_cluster.cc:1428] Setting key 3b5bae7a849f404c217df19d4ad227de
I20260619 06:38:37.123256 15556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:37.139024 15556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d: Bootstrap starting.
I20260619 06:38:37.139396 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:37.122730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.28.104.254:33215)
0619 06:38:37.124260 (+ 1530us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:37.124272 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:37.124313 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:38:37.126785 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:37.126814 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:37.127433 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:37.128543 (+ 1110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:37.129101 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:37.131133 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:37.133185 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:37.134222 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:37.134475 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:37.135866 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:37.135881 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:37.135899 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:37.136666 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:37.136764 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:37.136796 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:37.136890 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:37.137564 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 06:38:37.137956 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":505,"threads_started":1}
I20260619 06:38:37.143191 15556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:37.144459 15556 log.cc:824] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:37.148025 15556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d: No bootstrap required, opened a new log
I20260619 06:38:37.156827 15556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d52e37e63dd408f95007aa3dcfa5b2d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33215 } }
I20260619 06:38:37.157244 15556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:37.157369 15556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d52e37e63dd408f95007aa3dcfa5b2d, State: Initialized, Role: FOLLOWER
I20260619 06:38:37.157920 15556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d52e37e63dd408f95007aa3dcfa5b2d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33215 } }
I20260619 06:38:37.158154 15556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:37.158306 15556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:37.158495 15556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:37.160574 15556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d52e37e63dd408f95007aa3dcfa5b2d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33215 } }
I20260619 06:38:37.161211 15556 leader_election.cc:302] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d52e37e63dd408f95007aa3dcfa5b2d; no voters:
I20260619 06:38:37.161787 15556 leader_election.cc:288] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:37.161957 15561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:37.162668 15561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [term 1 LEADER]: Becoming Leader. State: Replica: 7d52e37e63dd408f95007aa3dcfa5b2d, State: Running, Role: LEADER
I20260619 06:38:37.163184 15561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d52e37e63dd408f95007aa3dcfa5b2d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33215 } }
I20260619 06:38:37.163774 15556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:37.166381 15562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d52e37e63dd408f95007aa3dcfa5b2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d52e37e63dd408f95007aa3dcfa5b2d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33215 } } }
I20260619 06:38:37.166744 15562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:37.167269 15563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d52e37e63dd408f95007aa3dcfa5b2d. Latest consensus state: current_term: 1 leader_uuid: "7d52e37e63dd408f95007aa3dcfa5b2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d52e37e63dd408f95007aa3dcfa5b2d" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 33215 } } }
I20260619 06:38:37.167625 15563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:37.177580 15568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:37.182718 15568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:37.190816 15568 catalog_manager.cc:1347] Generated new cluster ID: 138b5429acdc4f479b11804404c19d68
I20260619 06:38:37.190922 15568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:37.210564 15568 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:37.212366 15568 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:37.224050 15568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d: Generated new TSK 0
I20260619 06:38:37.224967 15568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:38:37.238583 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33215
--builtin_ntp_servers=127.28.104.212:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:38:37.573236 15580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:37.573529 15580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:37.573624 15580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:37.580914 15580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:37.581074 15580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:37.581229 15580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:38:37.590610 15580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:37.591938 15580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:37.594223 15580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:37.607565 15586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:37.607631 15585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:37.607611 15588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:37.608335 15580 server_base.cc:1034] running on GCE node
I20260619 06:38:37.608917 15580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:37.609972 15580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:37.611212 15580 hybrid_clock.cc:648] HybridClock initialized: now 1781851117611137 us; error 65 us; skew 500 ppm
I20260619 06:38:37.611574 15580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:37.614005 15580 webserver.cc:460] Webserver started at http://127.28.104.193:46849/ using document root <none> and password file <none>
I20260619 06:38:37.614701 15580 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:37.614821 15580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:37.615144 15580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:37.618042 15580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b"
format_stamp: "Formatted at 2026-06-19 06:38:37 on dist-test-slave-nf4n"
server_key: "95c780dfaf635feef0d9754c59f1249e"
server_key_iv: "92e0d84fb174593f906564d071a04b08"
server_key_version: "encryptionkey@0"
I20260619 06:38:37.618831 15580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b"
format_stamp: "Formatted at 2026-06-19 06:38:37 on dist-test-slave-nf4n"
server_key: "95c780dfaf635feef0d9754c59f1249e"
server_key_iv: "92e0d84fb174593f906564d071a04b08"
server_key_version: "encryptionkey@0"
I20260619 06:38:37.623991 15580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 06:38:37.627499 15594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:37.628831 15580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:38:37.629074 15580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b"
format_stamp: "Formatted at 2026-06-19 06:38:37 on dist-test-slave-nf4n"
server_key: "95c780dfaf635feef0d9754c59f1249e"
server_key_iv: "92e0d84fb174593f906564d071a04b08"
server_key_version: "encryptionkey@0"
I20260619 06:38:37.629310 15580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:37.650934 15580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:37.651737 15580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:37.652035 15580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:37.653118 15580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:37.654521 15580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:37.654647 15580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:37.654773 15580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:37.654857 15580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:37.687785 15580 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:42055
I20260619 06:38:37.687873 15707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:42055 every 8 connection(s)
I20260619 06:38:37.689167 15580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:38:37.693580 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 15580
I20260619 06:38:37.693768 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:38:37.694195 29091 external_mini_cluster.cc:1428] Setting key bfedaaf5854975c4daf35f6673db0eb4
I20260619 06:38:37.696736 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33215
--builtin_ntp_servers=127.28.104.212:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:38:37.701814 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:37.691654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:40877 (local address 127.28.104.254:33215)
0619 06:38:37.692110 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:37.692116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:37.693439 (+ 1323us) server_negotiation.cc:407] Connection header received
0619 06:38:37.695402 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:37.695407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:37.695497 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:37.695615 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:37.696637 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:37.697504 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:37.699008 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:37.699422 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:37.699563 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:37.700925 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:37.700929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:37.700932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:37.700949 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:37.700989 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:37.701002 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:37.701126 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:37.701358 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 06:38:37.701461 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260619 06:38:37.702782 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:37.692297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:33215 (local address 127.28.104.193:40877)
0619 06:38:37.693276 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:37.693327 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:37.695143 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:37.695838 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:37.695847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:37.695932 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:37.696479 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:37.696495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:37.697656 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:37.697661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:37.698861 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:37.698873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:37.699646 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:37.699654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:37.699900 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:37.700610 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:38:37.700613 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:37.700651 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:38:37.700729 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:37.701124 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:37.701137 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:37.701147 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 06:38:37.701336 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 06:38:37.701730 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":248,"threads_started":1}
I20260619 06:38:37.704356 15708 heartbeater.cc:347] Connected to a master server at 127.28.104.254:33215
I20260619 06:38:37.704680 15708 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:37.705596 15708 heartbeater.cc:511] Master 127.28.104.254:33215 requested a full tablet report, sending...
I20260619 06:38:37.707634 15521 ts_manager.cc:194] Registered new tserver with Master: f5d0dc5a6c344f50b97dd351d31e3b5b (127.28.104.193:42055)
I20260619 06:38:37.709173 15521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.193:40877
W20260619 06:38:38.024677 15712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:38.024945 15712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:38.025118 15712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:38.032814 15712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:38.032939 15712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:38.033141 15712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:38:38.042328 15712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:38.043651 15712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:38.045571 15712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:38.058348 15718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:38.058321 15717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:38.058531 15720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:38.059331 15712 server_base.cc:1034] running on GCE node
I20260619 06:38:38.059983 15712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:38.060971 15712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:38.062218 15712 hybrid_clock.cc:648] HybridClock initialized: now 1781851118062182 us; error 31 us; skew 500 ppm
I20260619 06:38:38.062546 15712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:38.065156 15712 webserver.cc:460] Webserver started at http://127.28.104.194:44733/ using document root <none> and password file <none>
I20260619 06:38:38.065837 15712 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:38.065968 15712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:38.066270 15712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:38.069240 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "8fb4e13faca54129813a5d81df0a10d3"
format_stamp: "Formatted at 2026-06-19 06:38:38 on dist-test-slave-nf4n"
server_key: "94ed636608d4361819439f5a64f71254"
server_key_iv: "81d94ffe94b71c9a903c4e194660443f"
server_key_version: "encryptionkey@0"
I20260619 06:38:38.070063 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "8fb4e13faca54129813a5d81df0a10d3"
format_stamp: "Formatted at 2026-06-19 06:38:38 on dist-test-slave-nf4n"
server_key: "94ed636608d4361819439f5a64f71254"
server_key_iv: "81d94ffe94b71c9a903c4e194660443f"
server_key_version: "encryptionkey@0"
I20260619 06:38:38.075002 15712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:38:38.078341 15726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:38.079666 15712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:38:38.079859 15712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "8fb4e13faca54129813a5d81df0a10d3"
format_stamp: "Formatted at 2026-06-19 06:38:38 on dist-test-slave-nf4n"
server_key: "94ed636608d4361819439f5a64f71254"
server_key_iv: "81d94ffe94b71c9a903c4e194660443f"
server_key_version: "encryptionkey@0"
I20260619 06:38:38.080077 15712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:38.096449 15712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:38.097281 15712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:38.097573 15712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:38.098500 15712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:38.099893 15712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:38.100016 15712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:38.100129 15712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:38.100210 15712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:38.133208 15712 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:36339
I20260619 06:38:38.133289 15838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:36339 every 8 connection(s)
I20260619 06:38:38.134563 15712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:38:38.142032 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 15712
I20260619 06:38:38.142208 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:38:38.142778 29091 external_mini_cluster.cc:1428] Setting key bec7494c22fe1c323369b5704edd387e
I20260619 06:38:38.145810 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33215
--builtin_ntp_servers=127.28.104.212:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:38:38.147303 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.137144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:56055 (local address 127.28.104.254:33215)
0619 06:38:38.137309 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.137314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.138675 (+ 1361us) server_negotiation.cc:407] Connection header received
0619 06:38:38.140337 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.140341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.140405 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.140495 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:38.141575 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.142419 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.144375 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.144914 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.145010 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.146615 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:38.146619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:38.146623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:38.146639 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:38.146668 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:38.146679 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:38.146749 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.147004 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.147076 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:38:38.148729 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.137657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:33215 (local address 127.28.104.194:56055)
0619 06:38:38.138511 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:38.138549 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:38.140103 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:38.140693 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:38.140700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:38.140805 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:38.141422 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.141439 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.142556 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.142562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.144242 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.144255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.145151 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.145159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.145452 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.146269 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:38:38.146275 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:38.146329 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:38:38.146430 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:38.146773 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:38.146793 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:38.146805 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 06:38:38.146981 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 06:38:38.147399 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":251,"threads_started":1}
I20260619 06:38:38.150085 15839 heartbeater.cc:347] Connected to a master server at 127.28.104.254:33215
I20260619 06:38:38.150396 15839 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:38.151106 15839 heartbeater.cc:511] Master 127.28.104.254:33215 requested a full tablet report, sending...
I20260619 06:38:38.152431 15521 ts_manager.cc:194] Registered new tserver with Master: 8fb4e13faca54129813a5d81df0a10d3 (127.28.104.194:36339)
I20260619 06:38:38.153282 15521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.194:56055
W20260619 06:38:38.473199 15842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:38.473471 15842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:38.473558 15842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:38.480701 15842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:38.480798 15842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:38.480911 15842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:38:38.490187 15842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:33215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:38.491415 15842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:38.493235 15842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:38.505970 15847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:38.506031 15848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:38.506099 15850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:38.507082 15842 server_base.cc:1034] running on GCE node
I20260619 06:38:38.507682 15842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:38.508626 15842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:38.509851 15842 hybrid_clock.cc:648] HybridClock initialized: now 1781851118509802 us; error 42 us; skew 500 ppm
I20260619 06:38:38.510182 15842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:38.512403 15842 webserver.cc:460] Webserver started at http://127.28.104.195:37199/ using document root <none> and password file <none>
I20260619 06:38:38.513144 15842 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:38.513260 15842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:38.513573 15842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:38.516412 15842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "515ddfb735c446ac96c14e3aaada8a93"
format_stamp: "Formatted at 2026-06-19 06:38:38 on dist-test-slave-nf4n"
server_key: "e03e9066b9845a70c62ee07af783029b"
server_key_iv: "38a2b7d8a301bd5c3818fc4c982d3be1"
server_key_version: "encryptionkey@0"
I20260619 06:38:38.517262 15842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "515ddfb735c446ac96c14e3aaada8a93"
format_stamp: "Formatted at 2026-06-19 06:38:38 on dist-test-slave-nf4n"
server_key: "e03e9066b9845a70c62ee07af783029b"
server_key_iv: "38a2b7d8a301bd5c3818fc4c982d3be1"
server_key_version: "encryptionkey@0"
I20260619 06:38:38.522275 15842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:38:38.525708 15856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:38.527060 15842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:38:38.527223 15842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "515ddfb735c446ac96c14e3aaada8a93"
format_stamp: "Formatted at 2026-06-19 06:38:38 on dist-test-slave-nf4n"
server_key: "e03e9066b9845a70c62ee07af783029b"
server_key_iv: "38a2b7d8a301bd5c3818fc4c982d3be1"
server_key_version: "encryptionkey@0"
I20260619 06:38:38.527438 15842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:38.544819 15842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:38.545714 15842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:38.546031 15842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:38.547014 15842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:38.548421 15842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:38.548543 15842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:38.548662 15842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:38.548750 15842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:38.582003 15842 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:39927
I20260619 06:38:38.582086 15969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:39927 every 8 connection(s)
I20260619 06:38:38.583446 15842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:38:38.590749 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 15842
I20260619 06:38:38.590945 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:38:38.591382 29091 external_mini_cluster.cc:1428] Setting key ca14ba4c93ae705aec04ca50dda928b1
I20260619 06:38:38.596619 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.586113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:39701 (local address 127.28.104.254:33215)
0619 06:38:38.586284 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.586289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.587652 (+ 1363us) server_negotiation.cc:407] Connection header received
0619 06:38:38.589442 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.589448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.589537 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.589650 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:38.590927 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.591776 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.593900 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.594437 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.594536 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.595850 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:38.595855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:38.595858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:38.595876 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:38.595906 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:38.595918 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:38.595990 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.596303 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.596383 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:38:38.597899 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.586617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:33215 (local address 127.28.104.195:39701)
0619 06:38:38.587499 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:38.587534 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:38.589180 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:38.589879 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:38.589888 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:38.589973 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:38.590714 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.590744 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.591921 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.591929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.593758 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.593772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.594618 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.594623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.594855 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.595555 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:38:38.595558 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:38.595599 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:38:38.595675 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:38.596036 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:38.596046 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:38.596053 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:38:38.596236 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 06:38:38.596553 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":259,"threads_started":1}
I20260619 06:38:38.599256 15970 heartbeater.cc:347] Connected to a master server at 127.28.104.254:33215
I20260619 06:38:38.599572 15970 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:38.600358 15970 heartbeater.cc:511] Master 127.28.104.254:33215 requested a full tablet report, sending...
I20260619 06:38:38.601727 15521 ts_manager.cc:194] Registered new tserver with Master: 515ddfb735c446ac96c14e3aaada8a93 (127.28.104.195:39927)
I20260619 06:38:38.602473 15521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.195:39701
I20260619 06:38:38.606477 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:38:38.616065 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.610467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.28.104.254:33215)
0619 06:38:38.610652 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.610657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.611073 (+ 416us) server_negotiation.cc:407] Connection header received
0619 06:38:38.611207 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.611211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.611280 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.611363 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:38.611780 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.612458 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.613819 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.614223 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.614315 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.615350 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:38.615355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:38.615358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:38.615375 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:38.615408 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:38.615421 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:38.615568 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.615749 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.615806 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:38:38.619740 15521 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:38:38.626642 15521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:38:38.655436 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.638707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:42055 (local address 127.0.0.1:54896)
0619 06:38:38.639462 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:38.639504 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:38.639703 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:38.648108 (+ 8405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:38.648115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:38.648154 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:38.648523 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.648534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.650440 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.650448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.652662 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.652676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.654479 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.654486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.654685 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.654737 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 06:38:38.654877 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 06:38:38.655068 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":4,"thread_start_us":433,"threads_started":1}
I20260619 06:38:38.655422 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.640043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.28.104.193:42055)
0619 06:38:38.646714 (+ 6671us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.646728 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.646770 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:38:38.646899 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.646911 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.647272 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.648758 (+ 1486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:38.649013 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.650254 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.652857 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.654288 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.654455 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.654707 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.654921 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.655093 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6495,"thread_start_us":6399,"threads_started":1}
I20260619 06:38:38.656865 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.639105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:36339 (local address 127.0.0.1:52764)
0619 06:38:38.645651 (+ 6546us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:38.645678 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:38.645841 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:38.648520 (+ 2679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:38.648525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:38.648549 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:38.648974 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.648988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.651212 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.651218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.653790 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.653807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.655833 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.655840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.656136 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.656279 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 06:38:38.656356 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:38:38.656496 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6291,"mutex_wait_us":459,"thread_start_us":6137,"threads_started":1}
I20260619 06:38:38.656993 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.639470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.28.104.195:39927)
0619 06:38:38.648274 (+ 8804us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.648283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.648314 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:38:38.648402 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.648412 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.648618 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.648813 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:38.649455 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.651033 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.653834 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.655252 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.655467 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.655653 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.656462 (+ 809us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.656618 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8651,"thread_start_us":8516,"threads_started":1}
I20260619 06:38:38.657619 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.639674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.28.104.194:36339)
0619 06:38:38.647891 (+ 8217us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.647903 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.647939 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:38:38.648049 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.648058 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.648313 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.648515 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:38.649495 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.651033 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.653979 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.655310 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.655455 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.655653 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.657172 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.657356 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8060,"thread_start_us":7939,"threads_started":1}
I20260619 06:38:38.658077 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.639140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:39927 (local address 127.0.0.1:51442)
0619 06:38:38.646400 (+ 7260us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:38.646421 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:38.646536 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:38.648809 (+ 2273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:38.648814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:38.648838 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:38.649275 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.649285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.651212 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.651218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.653642 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.653656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.655834 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.655840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.656136 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.656281 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 06:38:38.656356 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 06:38:38.657853 (+ 1497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7128,"mutex_wait_us":2367,"thread_start_us":6148,"threads_started":1}
I20260619 06:38:38.659651 15643 tablet_service.cc:1467] Processing CreateTablet for tablet 71190d41bd1f401a8c9fc60db4a7b2c9 (DEFAULT_TABLE table=test-table [id=9c2623d63ac74918b338731c5d0cd227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:38.660821 15643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71190d41bd1f401a8c9fc60db4a7b2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:38.660594 15774 tablet_service.cc:1467] Processing CreateTablet for tablet 71190d41bd1f401a8c9fc60db4a7b2c9 (DEFAULT_TABLE table=test-table [id=9c2623d63ac74918b338731c5d0cd227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:38.661852 15774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71190d41bd1f401a8c9fc60db4a7b2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:38.664598 15905 tablet_service.cc:1467] Processing CreateTablet for tablet 71190d41bd1f401a8c9fc60db4a7b2c9 (DEFAULT_TABLE table=test-table [id=9c2623d63ac74918b338731c5d0cd227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:38.666440 15905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71190d41bd1f401a8c9fc60db4a7b2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:38.675520 15990 tablet_bootstrap.cc:492] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b: Bootstrap starting.
I20260619 06:38:38.677323 15989 tablet_bootstrap.cc:492] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: Bootstrap starting.
I20260619 06:38:38.679373 15990 tablet_bootstrap.cc:654] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:38.680389 15990 log.cc:824] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:38.680871 15989 tablet_bootstrap.cc:654] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:38.682200 15991 tablet_bootstrap.cc:492] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: Bootstrap starting.
I20260619 06:38:38.682582 15989 log.cc:824] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:38.683650 15990 tablet_bootstrap.cc:492] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b: No bootstrap required, opened a new log
I20260619 06:38:38.683938 15990 ts_tablet_manager.cc:1397] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260619 06:38:38.685561 15989 tablet_bootstrap.cc:492] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: No bootstrap required, opened a new log
I20260619 06:38:38.685629 15991 tablet_bootstrap.cc:654] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:38.685837 15989 ts_tablet_manager.cc:1397] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:38:38.686770 15991 log.cc:824] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:38.691515 15991 tablet_bootstrap.cc:492] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: No bootstrap required, opened a new log
I20260619 06:38:38.691921 15991 ts_tablet_manager.cc:1397] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 06:38:38.695914 15989 raft_consensus.cc:348] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42055 } } peers { permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 } } peers { permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 } }
I20260619 06:38:38.696503 15989 raft_consensus.cc:374] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:38.696652 15989 raft_consensus.cc:729] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb4e13faca54129813a5d81df0a10d3, State: Initialized, Role: FOLLOWER
I20260619 06:38:38.697458 15989 consensus_queue.cc:260] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42055 } } peers { permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 } } peers { permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 } }
I20260619 06:38:38.697726 15990 raft_consensus.cc:348] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42055 } } peers { permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 } } peers { permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 } }
I20260619 06:38:38.698235 15990 raft_consensus.cc:374] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:38.698391 15990 raft_consensus.cc:729] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d0dc5a6c344f50b97dd351d31e3b5b, State: Initialized, Role: FOLLOWER
I20260619 06:38:38.699169 15990 consensus_queue.cc:260] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42055 } } peers { permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 } } peers { permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 } }
I20260619 06:38:38.702302 15991 raft_consensus.cc:348] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42055 } } peers { permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 } } peers { permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 } }
I20260619 06:38:38.702775 15991 raft_consensus.cc:374] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:38.702911 15991 raft_consensus.cc:729] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515ddfb735c446ac96c14e3aaada8a93, State: Initialized, Role: FOLLOWER
I20260619 06:38:38.703547 15991 consensus_queue.cc:260] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42055 } } peers { permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 } } peers { permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 } }
I20260619 06:38:38.705410 15839 heartbeater.cc:503] Master 127.28.104.254:33215 was elected leader, sending a full tablet report...
I20260619 06:38:38.705920 15989 ts_tablet_manager.cc:1428] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
I20260619 06:38:38.706032 15970 heartbeater.cc:503] Master 127.28.104.254:33215 was elected leader, sending a full tablet report...
I20260619 06:38:38.708393 15991 ts_tablet_manager.cc:1428] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: Time spent starting tablet: real 0.016s user 0.007s sys 0.005s
I20260619 06:38:38.708451 15990 ts_tablet_manager.cc:1428] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b: Time spent starting tablet: real 0.024s user 0.015s sys 0.007s
I20260619 06:38:38.709084 15708 heartbeater.cc:503] Master 127.28.104.254:33215 was elected leader, sending a full tablet report...
I20260619 06:38:38.760937 15996 raft_consensus.cc:482] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:38:38.761286 15996 raft_consensus.cc:504] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42055 } } peers { permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 } } peers { permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 } }
I20260619 06:38:38.762941 15996 leader_election.cc:288] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fb4e13faca54129813a5d81df0a10d3 (127.28.104.194:36339), 515ddfb735c446ac96c14e3aaada8a93 (127.28.104.195:39927)
I20260619 06:38:38.770987 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.763415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:35937 (local address 127.28.104.195:39927)
0619 06:38:38.763777 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.763796 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.764394 (+ 598us) server_negotiation.cc:407] Connection header received
0619 06:38:38.764506 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.764511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.764577 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.764664 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:38.765547 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.766583 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.769094 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.769967 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.770139 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.770222 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.770601 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.770728 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260619 06:38:38.770990 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.763529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:54523 (local address 127.28.104.194:36339)
0619 06:38:38.763754 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.763762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.764145 (+ 383us) server_negotiation.cc:407] Connection header received
0619 06:38:38.764467 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.764475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.764555 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.764664 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:38.765439 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.766558 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.769124 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.769954 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.770149 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.770224 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.770590 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.770715 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:38:38.771143 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.763370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:39927 (local address 127.28.104.193:35937)
0619 06:38:38.764282 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:38.764298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:38.764407 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:38.764703 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:38.764709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:38.764768 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:38.765383 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.765396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.766751 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.766754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.768885 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.768898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.770199 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.770207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.770453 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.770476 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:38:38.770576 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:38:38.770699 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":95,"spinlock_wait_cycles":5504,"thread_start_us":545,"threads_started":1}
I20260619 06:38:38.771186 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.763232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:36339 (local address 127.28.104.193:54523)
0619 06:38:38.764013 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:38.764040 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:38.764278 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:38.764703 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:38.764709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:38.764772 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:38.765280 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.765300 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.766728 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.766733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.768915 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:38.768928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.770150 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.770156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:38.770422 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.770458 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:38:38.770559 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:38:38.770692 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":62,"spinlock_wait_cycles":768,"thread_start_us":377,"threads_started":1}
I20260619 06:38:38.772209 15925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71190d41bd1f401a8c9fc60db4a7b2c9" candidate_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515ddfb735c446ac96c14e3aaada8a93" is_pre_election: true
I20260619 06:38:38.772308 15794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71190d41bd1f401a8c9fc60db4a7b2c9" candidate_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb4e13faca54129813a5d81df0a10d3" is_pre_election: true
I20260619 06:38:38.772776 15925 raft_consensus.cc:2454] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d0dc5a6c344f50b97dd351d31e3b5b in term 0.
I20260619 06:38:38.772847 15794 raft_consensus.cc:2454] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d0dc5a6c344f50b97dd351d31e3b5b in term 0.
I20260619 06:38:38.773540 15598 leader_election.cc:302] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 515ddfb735c446ac96c14e3aaada8a93, 8fb4e13faca54129813a5d81df0a10d3, f5d0dc5a6c344f50b97dd351d31e3b5b; no voters:
I20260619 06:38:38.773940 15996 raft_consensus.cc:2781] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:38:38.774080 15996 raft_consensus.cc:482] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:38:38.774185 15996 raft_consensus.cc:3037] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:38.776337 15996 raft_consensus.cc:504] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42055 } } peers { permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 } } peers { permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 } }
I20260619 06:38:38.777094 15996 leader_election.cc:288] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [CANDIDATE]: Term 1 election: Requested vote from peers 8fb4e13faca54129813a5d81df0a10d3 (127.28.104.194:36339), 515ddfb735c446ac96c14e3aaada8a93 (127.28.104.195:39927)
I20260619 06:38:38.777602 15794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71190d41bd1f401a8c9fc60db4a7b2c9" candidate_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb4e13faca54129813a5d81df0a10d3"
I20260619 06:38:38.777845 15794 raft_consensus.cc:3037] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:38.778890 15925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71190d41bd1f401a8c9fc60db4a7b2c9" candidate_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515ddfb735c446ac96c14e3aaada8a93"
I20260619 06:38:38.779101 15925 raft_consensus.cc:3037] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:38.781091 15794 raft_consensus.cc:2454] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d0dc5a6c344f50b97dd351d31e3b5b in term 1.
I20260619 06:38:38.781512 15598 leader_election.cc:302] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fb4e13faca54129813a5d81df0a10d3, f5d0dc5a6c344f50b97dd351d31e3b5b; no voters:
I20260619 06:38:38.781838 15996 raft_consensus.cc:2781] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:38.782331 15996 raft_consensus.cc:686] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 1 LEADER]: Becoming Leader. State: Replica: f5d0dc5a6c344f50b97dd351d31e3b5b, State: Running, Role: LEADER
I20260619 06:38:38.782343 15925 raft_consensus.cc:2454] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d0dc5a6c344f50b97dd351d31e3b5b in term 1.
I20260619 06:38:38.782944 15996 consensus_queue.cc:237] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42055 } } peers { permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 } } peers { permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 } }
I20260619 06:38:38.787473 15519 catalog_manager.cc:5495] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b reported cstate change: term changed from 0 to 1, leader changed from <none> to f5d0dc5a6c344f50b97dd351d31e3b5b (127.28.104.193). New cstate: current_term: 1 leader_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d0dc5a6c344f50b97dd351d31e3b5b" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 } health_report { overall_health: UNKNOWN } } }
I20260619 06:38:38.808795 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.803867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.28.104.193:42055)
0619 06:38:38.804078 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.804084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.804108 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:38:38.804239 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.804243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.804332 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.804456 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:38:38.804771 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.805508 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.806800 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.807195 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.807286 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.807728 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:38:38.808311 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:38:38.808443 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.808546 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.808581 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:38:38.817087 15925 raft_consensus.cc:1261] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93 [term 1 FOLLOWER]: Refusing update from remote peer f5d0dc5a6c344f50b97dd351d31e3b5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:38.817082 15794 raft_consensus.cc:1261] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3 [term 1 FOLLOWER]: Refusing update from remote peer f5d0dc5a6c344f50b97dd351d31e3b5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:38.818130 16000 consensus_queue.cc:1035] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "515ddfb735c446ac96c14e3aaada8a93" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:38.818476 15996 consensus_queue.cc:1035] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fb4e13faca54129813a5d81df0a10d3" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:38.842423 15519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:38.842726 15519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:38.846691 15519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d52e37e63dd408f95007aa3dcfa5b2d: Sending DeleteTablet for 3 replicas of tablet 71190d41bd1f401a8c9fc60db4a7b2c9
I20260619 06:38:38.847882 15774 tablet_service.cc:1514] Processing DeleteTablet for tablet 71190d41bd1f401a8c9fc60db4a7b2c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:38 UTC) from {username='slave'} at 127.0.0.1:52764
I20260619 06:38:38.847888 15643 tablet_service.cc:1514] Processing DeleteTablet for tablet 71190d41bd1f401a8c9fc60db4a7b2c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:38 UTC) from {username='slave'} at 127.0.0.1:54896
I20260619 06:38:38.848670 15905 tablet_service.cc:1514] Processing DeleteTablet for tablet 71190d41bd1f401a8c9fc60db4a7b2c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:38 UTC) from {username='slave'} at 127.0.0.1:51442
I20260619 06:38:38.850927 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 15580
I20260619 06:38:38.859942 16013 tablet_replica.cc:330] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b: stopping tablet replica
I20260619 06:38:38.860437 16013 raft_consensus.cc:2229] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:38:38.861181 16013 raft_consensus.cc:2258] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:38.862054 16012 tablet_replica.cc:330] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: stopping tablet replica
I20260619 06:38:38.862353 16014 tablet_replica.cc:330] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: stopping tablet replica
I20260619 06:38:38.862557 16012 raft_consensus.cc:2229] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:38.862952 16014 raft_consensus.cc:2229] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:38.863162 16012 raft_consensus.cc:2258] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:38.863607 16014 raft_consensus.cc:2258] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:38.866147 16012 ts_tablet_manager.cc:1905] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:38.866535 16014 ts_tablet_manager.cc:1905] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:38.873576 16012 ts_tablet_manager.cc:1918] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:38.873852 16012 log.cc:1196] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/71190d41bd1f401a8c9fc60db4a7b2c9
I20260619 06:38:38.874256 16014 ts_tablet_manager.cc:1918] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:38.874500 16012 ts_tablet_manager.cc:1939] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 8fb4e13faca54129813a5d81df0a10d3: Deleting consensus metadata
I20260619 06:38:38.875896 16013 ts_tablet_manager.cc:1905] T 71190d41bd1f401a8c9fc60db4a7b2c9 P f5d0dc5a6c344f50b97dd351d31e3b5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:38.876564 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.853294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.28.104.193:42055)
0619 06:38:38.854199 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.854208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.867667 (+ 13459us) server_negotiation.cc:407] Connection header received
0619 06:38:38.868158 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.868169 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.868297 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.868373 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:38.868999 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.870245 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.872359 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.873079 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.873179 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.874767 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:38.874778 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:38.874798 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:38.875633 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:38.875700 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:38.875724 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:38.875807 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.876066 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.876172 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730}
I20260619 06:38:38.876502 15508 catalog_manager.cc:4841] TS 8fb4e13faca54129813a5d81df0a10d3 (127.28.104.194:36339): tablet 71190d41bd1f401a8c9fc60db4a7b2c9 (table test-table [id=9c2623d63ac74918b338731c5d0cd227]) successfully deleted
I20260619 06:38:38.878106 15703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:38.878589 16014 log.cc:1196] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/71190d41bd1f401a8c9fc60db4a7b2c9
I20260619 06:38:38.879262 16014 ts_tablet_manager.cc:1939] T 71190d41bd1f401a8c9fc60db4a7b2c9 P 515ddfb735c446ac96c14e3aaada8a93: Deleting consensus metadata
I20260619 06:38:38.881588 15507 catalog_manager.cc:4841] TS 515ddfb735c446ac96c14e3aaada8a93 (127.28.104.195:39927): tablet 71190d41bd1f401a8c9fc60db4a7b2c9 (table test-table [id=9c2623d63ac74918b338731c5d0cd227]) successfully deleted
I20260619 06:38:38.974858 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 15580
I20260619 06:38:38.993247 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 15712
W20260619 06:38:38.993227 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.104.193:42055 (error 108)
I20260619 06:38:38.995011 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.994599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:42055 (local address 127.0.0.1:54916)
0619 06:38:38.994763 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:38.994870 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:38:38.995240 15506 catalog_manager.cc:4584] TS f5d0dc5a6c344f50b97dd351d31e3b5b (127.28.104.193:42055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71190d41bd1f401a8c9fc60db4a7b2c9: Network error: Client connection negotiation failed: client connection to 127.28.104.193:42055: connect: Connection refused (error 111)
I20260619 06:38:39.000273 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:38.994106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.28.104.194:36339)
0619 06:38:38.994288 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:38.994294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:38.994429 (+ 135us) server_negotiation.cc:407] Connection header received
0619 06:38:38.994669 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:38.994673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:38.994755 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:38.994856 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:38.995284 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.996118 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.997473 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:38.997859 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:38.997954 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:38.999001 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:38.999009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:38.999023 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:38.999579 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:38.999629 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:38.999647 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:38.999732 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:38.999846 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 06:38:38.999936 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:38:39.000907 15834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:39.048696 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:39.048239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:42055 (local address 127.0.0.1:54932)
0619 06:38:39.048439 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:39.048542 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 06:38:39.049820 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:39.049494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:42055 (local address 127.0.0.1:54946)
0619 06:38:39.049631 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:39.049698 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:38:39.092741 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 15712
I20260619 06:38:39.109711 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 15842
I20260619 06:38:39.110934 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:39.110462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:42055 (local address 127.0.0.1:54950)
0619 06:38:39.110683 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:39.110770 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 06:38:39.111910 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:39.111590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:42055 (local address 127.0.0.1:54956)
0619 06:38:39.111730 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:39.111804 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 06:38:39.116791 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:39.110474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.28.104.195:39927)
0619 06:38:39.110643 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:39.110650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:39.110674 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:38:39.110918 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:39.110923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:39.111033 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:39.111127 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:39.111595 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:39.112424 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:39.113748 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:39.114120 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:39.114213 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:39.115176 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:39.115184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:39.115199 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:39.115764 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:39.115844 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:39.115869 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:39.115976 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:39.116082 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 06:38:39.116189 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:38:39.117511 15965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:39.192528 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:39.191965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:42055 (local address 127.0.0.1:54958)
0619 06:38:39.192197 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:39.192325 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:38:39.193667 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:39.193206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:42055 (local address 127.0.0.1:54974)
0619 06:38:39.193423 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:39.193542 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.193:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 06:38:39.207835 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 15842
I20260619 06:38:39.224275 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 15490
I20260619 06:38:39.225392 15551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:39.337057 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 15490
2026-06-19T06:38:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2731 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:38:39 dist-test-slave-nf4n krb5kdc[16023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:38:39 dist-test-slave-nf4n krb5kdc[16023](info): set up 2 sockets
Jun 19 06:38:39 dist-test-slave-nf4n krb5kdc[16023](info): commencing operation
krb5kdc: starting...
W20260619 06:38:40.444952 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:38:40 dist-test-slave-nf4n krb5kdc[16023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:38:40.689450 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:38715
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:38715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:41.017745 16036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:41.018041 16036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:41.018129 16036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:41.025324 16036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:41.025422 16036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:41.025475 16036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:41.025529 16036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:41.025573 16036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:41.035794 16036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:38715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:41.037066 16036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:41.038353 16036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:41.049886 16044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:41.049947 16041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:41.049968 16042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:41.050884 16036 server_base.cc:1034] running on GCE node
I20260619 06:38:41.051729 16036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:41.053125 16036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:41.054360 16036 hybrid_clock.cc:648] HybridClock initialized: now 1781851121054288 us; error 58 us; skew 500 ppm
I20260619 06:38:41.054694 16036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:41 dist-test-slave-nf4n krb5kdc[16023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851121, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:41.061339 16036 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:38:41.062765 16036 webserver.cc:460] Webserver started at http://127.28.104.254:43935/ using document root <none> and password file <none>
I20260619 06:38:41.063454 16036 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:41.063593 16036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:41.063891 16036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:41.067337 16036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "1a241489871b4a138ac7fe389a64ef5c"
format_stamp: "Formatted at 2026-06-19 06:38:41 on dist-test-slave-nf4n"
server_key: "ee473b700e5b7c25f42ea9f77f76ff5e"
server_key_iv: "6e10fe9cf044535feaf832cb601304c4"
server_key_version: "encryptionkey@0"
I20260619 06:38:41.068174 16036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "1a241489871b4a138ac7fe389a64ef5c"
format_stamp: "Formatted at 2026-06-19 06:38:41 on dist-test-slave-nf4n"
server_key: "ee473b700e5b7c25f42ea9f77f76ff5e"
server_key_iv: "6e10fe9cf044535feaf832cb601304c4"
server_key_version: "encryptionkey@0"
I20260619 06:38:41.073689 16036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:38:41.077518 16051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:41.078938 16036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:38:41.079171 16036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "1a241489871b4a138ac7fe389a64ef5c"
format_stamp: "Formatted at 2026-06-19 06:38:41 on dist-test-slave-nf4n"
server_key: "ee473b700e5b7c25f42ea9f77f76ff5e"
server_key_iv: "6e10fe9cf044535feaf832cb601304c4"
server_key_version: "encryptionkey@0"
I20260619 06:38:41.079372 16036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:41.098992 16036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:41.103273 16036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:41.103551 16036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:41.121953 16036 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:38715
I20260619 06:38:41.121971 16102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:38715 every 8 connection(s)
I20260619 06:38:41.123369 16036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:41.126957 16103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:41.132841 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 16036
I20260619 06:38:41.133060 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:41.133575 29091 external_mini_cluster.cc:1428] Setting key c46d115a2471560fde0483dd555cd574
I20260619 06:38:41.138017 16103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c: Bootstrap starting.
I20260619 06:38:41.141762 16103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:41.142948 16103 log.cc:824] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:41.146155 16103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c: No bootstrap required, opened a new log
Jun 19 06:38:41 dist-test-slave-nf4n krb5kdc[16023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:41.154207 16103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a241489871b4a138ac7fe389a64ef5c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38715 } }
I20260619 06:38:41.154592 16103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:41.154713 16103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a241489871b4a138ac7fe389a64ef5c, State: Initialized, Role: FOLLOWER
I20260619 06:38:41.155341 16103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a241489871b4a138ac7fe389a64ef5c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38715 } }
I20260619 06:38:41.155654 16103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:41.155802 16103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:41.155980 16103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:41.157217 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:41.135522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.28.104.254:38715)
0619 06:38:41.136352 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:41.136363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:41.136405 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:38:41.138148 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:41.138177 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:41.138583 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:41.138894 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:41.140695 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:41.142135 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:41.143862 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:41.144716 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:41.144860 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:41.149611 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:41.149623 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:41.149637 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:41.149672 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:41.152757 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:41.153481 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:41.153488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:41.153493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:41.153651 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:41.154024 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:41.154029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:41.154032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:41.154575 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:41.154891 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:41.155316 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 06:38:41.155805 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":628,"thread_start_us":392,"threads_started":1}
I20260619 06:38:41.158681 16103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a241489871b4a138ac7fe389a64ef5c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38715 } }
I20260619 06:38:41.159164 16103 leader_election.cc:302] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a241489871b4a138ac7fe389a64ef5c; no voters:
I20260619 06:38:41.159816 16103 leader_election.cc:288] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:41.159942 16109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:41.160599 16109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [term 1 LEADER]: Becoming Leader. State: Replica: 1a241489871b4a138ac7fe389a64ef5c, State: Running, Role: LEADER
I20260619 06:38:41.161264 16109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a241489871b4a138ac7fe389a64ef5c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38715 } }
I20260619 06:38:41.162068 16103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:41.164057 16110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a241489871b4a138ac7fe389a64ef5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a241489871b4a138ac7fe389a64ef5c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38715 } } }
I20260619 06:38:41.164403 16110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:41.164834 16111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a241489871b4a138ac7fe389a64ef5c. Latest consensus state: current_term: 1 leader_uuid: "1a241489871b4a138ac7fe389a64ef5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a241489871b4a138ac7fe389a64ef5c" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38715 } } }
I20260619 06:38:41.165202 16111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:41.174185 16115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:41.178241 16115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:41.186311 16115 catalog_manager.cc:1347] Generated new cluster ID: 1a524c360fb94c759489525ed8a6c3c1
I20260619 06:38:41.186444 16115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:41.202756 16115 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:41.204032 16115 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:41.218545 16115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c: Generated new TSK 0
I20260619 06:38:41.219393 16115 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.104.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
Entry for principal kudu/127.28.104.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab.
I20260619 06:38:41.274032 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:38715
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:41.601821 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:41.602130 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:41.602303 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:41.609535 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:41.609683 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:41.609871 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:38:41.620002 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.193.keytab
--principal=kudu/127.28.104.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:38715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:41.621263 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:41.622608 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:41.635193 16135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:41.635025 16136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:41.635026 16138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:41.635919 16130 server_base.cc:1034] running on GCE node
I20260619 06:38:41.636469 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:41.637430 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:41.638631 16130 hybrid_clock.cc:648] HybridClock initialized: now 1781851121638578 us; error 40 us; skew 500 ppm
I20260619 06:38:41.638980 16130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:41 dist-test-slave-nf4n krb5kdc[16023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851121, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:41.642211 16130 init.cc:377] Logged in from keytab as kudu/127.28.104.193@KRBTEST.COM (short username kudu)
I20260619 06:38:41.643537 16130 webserver.cc:460] Webserver started at http://127.28.104.193:43127/ using document root <none> and password file <none>
I20260619 06:38:41.644213 16130 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:41.644306 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:41.644687 16130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:41.647652 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "bab36a41c5c5405a95dab40b0a74ae58"
format_stamp: "Formatted at 2026-06-19 06:38:41 on dist-test-slave-nf4n"
server_key: "b17e264887a1b7ee0ecad390b44cb0fe"
server_key_iv: "c60158c4c5d39f9283981426af417654"
server_key_version: "encryptionkey@0"
I20260619 06:38:41.648413 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "bab36a41c5c5405a95dab40b0a74ae58"
format_stamp: "Formatted at 2026-06-19 06:38:41 on dist-test-slave-nf4n"
server_key: "b17e264887a1b7ee0ecad390b44cb0fe"
server_key_iv: "c60158c4c5d39f9283981426af417654"
server_key_version: "encryptionkey@0"
I20260619 06:38:41.653692 16130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 06:38:41.657434 16145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:41.658776 16130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:38:41.658946 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "bab36a41c5c5405a95dab40b0a74ae58"
format_stamp: "Formatted at 2026-06-19 06:38:41 on dist-test-slave-nf4n"
server_key: "b17e264887a1b7ee0ecad390b44cb0fe"
server_key_iv: "c60158c4c5d39f9283981426af417654"
server_key_version: "encryptionkey@0"
I20260619 06:38:41.659129 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:41.678310 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:41.682456 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:41.682801 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:41.683717 16130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:41.685135 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:41.685210 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:41.685360 16130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:41.685437 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:41.716691 16130 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:42617
I20260619 06:38:41.716730 16258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:42617 every 8 connection(s)
I20260619 06:38:41.718001 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:38:41.721866 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 16130
I20260619 06:38:41.722048 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:38:41.722491 29091 external_mini_cluster.cc:1428] Setting key 9b540c62ad8b9dc424e0f9ba9e669ad4
Jun 19 06:38:41 dist-test-slave-nf4n krb5kdc[16023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851121, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.193@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:41.739321 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:41.720543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:56233 (local address 127.28.104.254:38715)
0619 06:38:41.721097 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:41.721103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:41.721856 (+ 753us) server_negotiation.cc:407] Connection header received
0619 06:38:41.723937 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:41.723942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:41.724025 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:41.724151 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:41.725888 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:41.726607 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:41.728111 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:41.728522 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:41.728651 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:41.732343 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:41.732353 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:41.732358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:41.732414 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:41.735887 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:41.736606 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:41.736615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:41.736621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:41.736779 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:41.737256 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:41.737264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:41.737269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:41.737711 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:41.737863 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:41.738562 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 06:38:41.738829 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":294,"threads_started":1}
I20260619 06:38:41.739908 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:41.720950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:38715 (local address 127.28.104.193:56233)
0619 06:38:41.721692 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:41.721739 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:41.723671 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:41.724600 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:41.724608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:41.725167 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:41.725720 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:41.725737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:41.726756 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:41.726760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:41.727973 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:41.727984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:41.728754 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:41.728761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:41.728986 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:41.729698 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:38:41.729723 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:41.732098 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:41.736030 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:41.736037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:41.736050 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:41.736467 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:41.736908 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:41.736911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:41.736914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:41.737144 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:41.737844 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:41.737850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:41.738168 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 06:38:41.738401 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 06:38:41.738792 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":275,"threads_started":1}
I20260619 06:38:41.741516 16259 heartbeater.cc:347] Connected to a master server at 127.28.104.254:38715
I20260619 06:38:41.741860 16259 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:41.742739 16259 heartbeater.cc:511] Master 127.28.104.254:38715 requested a full tablet report, sending...
I20260619 06:38:41.744967 16068 ts_manager.cc:194] Registered new tserver with Master: bab36a41c5c5405a95dab40b0a74ae58 (127.28.104.193:42617)
I20260619 06:38:41.746501 16068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.193@KRBTEST.COM'} at 127.28.104.193:56233
WARNING: no policy specified for kudu/127.28.104.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
Entry for principal kudu/127.28.104.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab.
I20260619 06:38:41.779531 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:38715
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:42.104605 16265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:42.104940 16265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:42.105149 16265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:42.112450 16265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:42.112604 16265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:42.112795 16265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:38:42.123847 16265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.194.keytab
--principal=kudu/127.28.104.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:38715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:42.125406 16265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:42.127101 16265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:42.141866 16272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:42.142520 16271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:42.142892 16274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:42.143404 16265 server_base.cc:1034] running on GCE node
I20260619 06:38:42.143976 16265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:42.144905 16265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:42.146121 16265 hybrid_clock.cc:648] HybridClock initialized: now 1781851122146068 us; error 72 us; skew 500 ppm
I20260619 06:38:42.146416 16265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:42 dist-test-slave-nf4n krb5kdc[16023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851122, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:42.149547 16265 init.cc:377] Logged in from keytab as kudu/127.28.104.194@KRBTEST.COM (short username kudu)
I20260619 06:38:42.150853 16265 webserver.cc:460] Webserver started at http://127.28.104.194:40135/ using document root <none> and password file <none>
I20260619 06:38:42.151486 16265 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:42.151618 16265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:42.151918 16265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:42.154743 16265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "e8750beb01b443cfa153e6834e2bc4bc"
format_stamp: "Formatted at 2026-06-19 06:38:42 on dist-test-slave-nf4n"
server_key: "ec4e53bbb4596008c218fd2cb3d85024"
server_key_iv: "b35fc0d40fcc763ff3d785c17cec0b3e"
server_key_version: "encryptionkey@0"
I20260619 06:38:42.155467 16265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "e8750beb01b443cfa153e6834e2bc4bc"
format_stamp: "Formatted at 2026-06-19 06:38:42 on dist-test-slave-nf4n"
server_key: "ec4e53bbb4596008c218fd2cb3d85024"
server_key_iv: "b35fc0d40fcc763ff3d785c17cec0b3e"
server_key_version: "encryptionkey@0"
I20260619 06:38:42.160295 16265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 06:38:42.163717 16281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:42.164935 16265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:38:42.165127 16265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "e8750beb01b443cfa153e6834e2bc4bc"
format_stamp: "Formatted at 2026-06-19 06:38:42 on dist-test-slave-nf4n"
server_key: "ec4e53bbb4596008c218fd2cb3d85024"
server_key_iv: "b35fc0d40fcc763ff3d785c17cec0b3e"
server_key_version: "encryptionkey@0"
I20260619 06:38:42.165306 16265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:42.178354 16265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:42.182611 16265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:42.182979 16265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:42.183930 16265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:42.185231 16265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:42.185303 16265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:42.185459 16265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:42.185523 16265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:42.217507 16265 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:36163
I20260619 06:38:42.217550 16394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:36163 every 8 connection(s)
I20260619 06:38:42.218829 16265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:38:42.223912 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 16265
I20260619 06:38:42.224102 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:38:42.224617 29091 external_mini_cluster.cc:1428] Setting key c66479919e734a22e832d70699f27a0e
Jun 19 06:38:42 dist-test-slave-nf4n krb5kdc[16023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851122, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.194@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:42.241473 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.221343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:38369 (local address 127.28.104.254:38715)
0619 06:38:42.221512 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:42.221517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:42.222698 (+ 1181us) server_negotiation.cc:407] Connection header received
0619 06:38:42.225107 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:42.225112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:42.225193 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:42.225343 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:42.227136 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.227832 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.230369 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.230781 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.230876 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.235156 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:42.235162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:42.235166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:42.235200 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:42.238113 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:42.238932 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:42.238939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:42.238943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:42.239104 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:42.239566 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:42.239573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:42.239576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:42.239877 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:42.240043 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:42.240788 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 06:38:42.241066 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:38:42.242513 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.221842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:38715 (local address 127.28.104.194:38369)
0619 06:38:42.222539 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:42.222582 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:42.224751 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:42.225615 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:42.225627 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:42.226167 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:42.226958 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.226987 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.227983 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.227992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.230205 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.230222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.230999 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.231008 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.231332 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.232061 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:38:42.232095 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:42.234897 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:42.238280 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:42.238291 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:42.238308 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:42.238789 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:42.239223 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:42.239230 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:42.239234 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:42.239446 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:42.240014 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:42.240022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:42.240411 (+ 389us) client_negotiation.cc:769] Sending connection context
0619 06:38:42.240706 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 06:38:42.241316 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":269,"threads_started":1}
I20260619 06:38:42.244181 16395 heartbeater.cc:347] Connected to a master server at 127.28.104.254:38715
I20260619 06:38:42.244504 16395 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:42.245286 16395 heartbeater.cc:511] Master 127.28.104.254:38715 requested a full tablet report, sending...
I20260619 06:38:42.247040 16068 ts_manager.cc:194] Registered new tserver with Master: e8750beb01b443cfa153e6834e2bc4bc (127.28.104.194:36163)
I20260619 06:38:42.247864 16068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.194@KRBTEST.COM'} at 127.28.104.194:38369
WARNING: no policy specified for kudu/127.28.104.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
Entry for principal kudu/127.28.104.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab.
I20260619 06:38:42.279675 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:38715
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:42.600184 16400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:42.600466 16400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:42.600560 16400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:42.607549 16400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:42.607654 16400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:42.607765 16400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:38:42.617242 16400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.195.keytab
--principal=kudu/127.28.104.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:38715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:42.618451 16400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:42.619707 16400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:42.632977 16408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:42.633137 16405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:42.633190 16406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:42.634132 16400 server_base.cc:1034] running on GCE node
I20260619 06:38:42.634727 16400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:42.635636 16400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:42.636830 16400 hybrid_clock.cc:648] HybridClock initialized: now 1781851122636787 us; error 33 us; skew 500 ppm
I20260619 06:38:42.637147 16400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:42 dist-test-slave-nf4n krb5kdc[16023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851122, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:42.640089 16400 init.cc:377] Logged in from keytab as kudu/127.28.104.195@KRBTEST.COM (short username kudu)
I20260619 06:38:42.641539 16400 webserver.cc:460] Webserver started at http://127.28.104.195:41399/ using document root <none> and password file <none>
I20260619 06:38:42.642217 16400 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:42.642318 16400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:42.642555 16400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:42.645315 16400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "4fbd8ab31f454058830d99dd83520585"
format_stamp: "Formatted at 2026-06-19 06:38:42 on dist-test-slave-nf4n"
server_key: "cfe1a174fff056302cb8bcdeaf529794"
server_key_iv: "4f38f5cfdb910d365a1a7523246af206"
server_key_version: "encryptionkey@0"
I20260619 06:38:42.646000 16400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "4fbd8ab31f454058830d99dd83520585"
format_stamp: "Formatted at 2026-06-19 06:38:42 on dist-test-slave-nf4n"
server_key: "cfe1a174fff056302cb8bcdeaf529794"
server_key_iv: "4f38f5cfdb910d365a1a7523246af206"
server_key_version: "encryptionkey@0"
I20260619 06:38:42.650833 16400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:38:42.654109 16415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:42.655328 16400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:38:42.655486 16400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "4fbd8ab31f454058830d99dd83520585"
format_stamp: "Formatted at 2026-06-19 06:38:42 on dist-test-slave-nf4n"
server_key: "cfe1a174fff056302cb8bcdeaf529794"
server_key_iv: "4f38f5cfdb910d365a1a7523246af206"
server_key_version: "encryptionkey@0"
I20260619 06:38:42.655663 16400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:42.677210 16400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:42.681002 16400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:42.681298 16400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:42.682293 16400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:42.683688 16400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:42.683760 16400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:42.683849 16400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:42.683892 16400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:42.716308 16400 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:44453
I20260619 06:38:42.716398 16528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:44453 every 8 connection(s)
I20260619 06:38:42.717980 16400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:38:42.723266 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 16400
I20260619 06:38:42.723465 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:38:42.723992 29091 external_mini_cluster.cc:1428] Setting key e5cb8b5ed5da7c1a069296f48578bdbe
Jun 19 06:38:42 dist-test-slave-nf4n krb5kdc[16023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851122, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.195@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:38:42.738307 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.720720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:57521 (local address 127.28.104.254:38715)
0619 06:38:42.720904 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:42.720908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:42.722087 (+ 1179us) server_negotiation.cc:407] Connection header received
0619 06:38:42.724518 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:42.724524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:42.724622 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:42.724740 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:42.726623 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.727433 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.729004 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.729437 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.729530 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.732833 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:42.732837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:42.732841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:42.732871 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:42.735371 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:42.736013 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:42.736017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:42.736019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:42.736135 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:42.736512 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:42.736515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:42.736517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:42.736758 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:42.736908 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:42.737793 (+ 885us) server_negotiation.cc:299] Negotiation successful
0619 06:38:42.738028 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:38:42.739518 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.721116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:38715 (local address 127.28.104.195:57521)
0619 06:38:42.721913 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:42.721950 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:42.724177 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:42.725016 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:42.725046 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:42.725814 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:42.726442 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.726464 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.727580 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.727584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.728890 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.728902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.729611 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.729615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.729811 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.730482 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:38:42.730506 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:42.732633 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:42.735496 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:42.735502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:42.735514 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:42.735921 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:42.736218 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:38:42.736221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:38:42.736223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:38:42.736421 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:38:42.736867 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:42.736872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:42.737346 (+ 474us) client_negotiation.cc:769] Sending connection context
0619 06:38:42.737644 (+ 298us) client_negotiation.cc:240] Negotiation successful
0619 06:38:42.738176 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":309,"threads_started":1}
I20260619 06:38:42.740823 16529 heartbeater.cc:347] Connected to a master server at 127.28.104.254:38715
I20260619 06:38:42.741163 16529 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:42.741876 16529 heartbeater.cc:511] Master 127.28.104.254:38715 requested a full tablet report, sending...
I20260619 06:38:42.743400 16068 ts_manager.cc:194] Registered new tserver with Master: 4fbd8ab31f454058830d99dd83520585 (127.28.104.195:44453)
I20260619 06:38:42.744251 16068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.104.195@KRBTEST.COM'} at 127.28.104.195:57521
I20260619 06:38:42.750048 16259 heartbeater.cc:503] Master 127.28.104.254:38715 was elected leader, sending a full tablet report...
I20260619 06:38:42.751410 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:38:42.768150 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.755454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.28.104.254:38715)
0619 06:38:42.755656 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:42.755661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:42.756156 (+ 495us) server_negotiation.cc:407] Connection header received
0619 06:38:42.756298 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:42.756302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:42.756360 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:42.756443 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:42.757943 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.758922 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.760347 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.760816 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.760913 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.763238 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:42.763243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:42.763246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:42.763276 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:42.765941 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:42.766523 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:42.766526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:42.766529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:42.766655 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:42.766964 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:42.766967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:42.766968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:42.767255 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:42.767435 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:42.767679 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 06:38:42.767850 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 06:38:42.771996 16068 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:38:42.781612 16068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:38:42.804260 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.793567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.28.104.194:36163)
0619 06:38:42.796600 (+ 3033us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:42.796609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:42.796635 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:38:42.796724 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:42.796732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:42.797022 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:42.797970 (+ 948us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:42.798175 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.799669 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.802098 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.803011 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.803243 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.803425 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:42.803749 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 06:38:42.803922 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2867,"thread_start_us":2733,"threads_started":1}
I20260619 06:38:42.804323 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.793392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:36163 (local address 127.0.0.1:47436)
0619 06:38:42.794434 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:42.794490 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:42.794702 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:42.797267 (+ 2565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:42.797273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:42.797300 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:42.797733 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.797745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.799816 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.799822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.801962 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.801972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.803275 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.803292 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.803493 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.803553 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 06:38:42.803689 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 06:38:42.803868 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":69,"thread_start_us":1681,"threads_started":1}
I20260619 06:38:42.805150 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.793690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:42617 (local address 127.0.0.1:42776)
0619 06:38:42.795338 (+ 1648us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:42.795357 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:42.795475 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:42.796438 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:42.796445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:42.796475 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:42.796856 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.796868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.799195 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.799203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.801525 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.801541 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.803405 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.803411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.803565 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.803589 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:38:42.803665 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:38:42.804738 (+ 1073us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1531,"mutex_wait_us":896,"thread_start_us":934,"threads_started":1}
I20260619 06:38:42.805641 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.794372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.28.104.193:42617)
0619 06:38:42.795783 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:42.795794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:42.795823 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:38:42.795916 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:42.795923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:42.796170 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:42.797109 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:42.797412 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.799044 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.801695 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.803190 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.803872 (+ 682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.805087 (+ 1215us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:42.805162 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 06:38:42.805330 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1222,"thread_start_us":1390,"threads_started":1}
I20260619 06:38:42.805716 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.795581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:44453 (local address 127.0.0.1:48692)
0619 06:38:42.798076 (+ 2495us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:42.798093 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:42.798277 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:42.798746 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:42.798750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:42.798782 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:42.799094 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.799102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.800723 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.800732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.802977 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.802993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.804925 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.804932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.805133 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.805168 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:38:42.805253 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:38:42.805362 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2356,"mutex_wait_us":20,"thread_start_us":2228,"threads_started":1}
I20260619 06:38:42.806382 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.796064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.28.104.195:44453)
0619 06:38:42.797602 (+ 1538us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:42.797613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:42.798177 (+ 564us) server_negotiation.cc:407] Connection header received
0619 06:38:42.798438 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:42.798444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:42.798589 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:42.798756 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:42.799310 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.800572 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.803143 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.804432 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.804607 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.804763 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:42.805981 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0619 06:38:42.806149 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1334,"thread_start_us":1168,"threads_started":1}
I20260619 06:38:42.808735 16329 tablet_service.cc:1467] Processing CreateTablet for tablet 92fff1643ba54a5bab97820388c89cb7 (DEFAULT_TABLE table=test-table [id=ebd4e131566e40fb91081c5d2a101c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:42.809095 16193 tablet_service.cc:1467] Processing CreateTablet for tablet 92fff1643ba54a5bab97820388c89cb7 (DEFAULT_TABLE table=test-table [id=ebd4e131566e40fb91081c5d2a101c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:42.809355 16464 tablet_service.cc:1467] Processing CreateTablet for tablet 92fff1643ba54a5bab97820388c89cb7 (DEFAULT_TABLE table=test-table [id=ebd4e131566e40fb91081c5d2a101c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:42.810253 16329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fff1643ba54a5bab97820388c89cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:42.810393 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fff1643ba54a5bab97820388c89cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:42.810611 16464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fff1643ba54a5bab97820388c89cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:42.821457 16547 tablet_bootstrap.cc:492] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: Bootstrap starting.
I20260619 06:38:42.824602 16548 tablet_bootstrap.cc:492] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: Bootstrap starting.
I20260619 06:38:42.824880 16547 tablet_bootstrap.cc:654] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:42.825654 16549 tablet_bootstrap.cc:492] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: Bootstrap starting.
I20260619 06:38:42.826759 16547 log.cc:824] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:42.829267 16548 tablet_bootstrap.cc:654] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:42.829409 16547 tablet_bootstrap.cc:492] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: No bootstrap required, opened a new log
I20260619 06:38:42.829661 16547 ts_tablet_manager.cc:1397] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 06:38:42.830476 16548 log.cc:824] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:42.830793 16549 tablet_bootstrap.cc:654] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:42.832257 16549 log.cc:824] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:42.835368 16549 tablet_bootstrap.cc:492] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: No bootstrap required, opened a new log
I20260619 06:38:42.835500 16548 tablet_bootstrap.cc:492] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: No bootstrap required, opened a new log
I20260619 06:38:42.835705 16549 ts_tablet_manager.cc:1397] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 06:38:42.835804 16548 ts_tablet_manager.cc:1397] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260619 06:38:42.839231 16547 raft_consensus.cc:348] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 } } peers { permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 } } peers { permanent_uuid: "4fbd8ab31f454058830d99dd83520585" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44453 } }
I20260619 06:38:42.839672 16547 raft_consensus.cc:374] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:42.839809 16547 raft_consensus.cc:729] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fbd8ab31f454058830d99dd83520585, State: Initialized, Role: FOLLOWER
I20260619 06:38:42.840444 16547 consensus_queue.cc:260] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 } } peers { permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 } } peers { permanent_uuid: "4fbd8ab31f454058830d99dd83520585" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44453 } }
I20260619 06:38:42.841575 16529 heartbeater.cc:503] Master 127.28.104.254:38715 was elected leader, sending a full tablet report...
I20260619 06:38:42.842250 16547 ts_tablet_manager.cc:1428] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260619 06:38:42.843717 16549 raft_consensus.cc:348] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 } } peers { permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 } } peers { permanent_uuid: "4fbd8ab31f454058830d99dd83520585" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44453 } }
I20260619 06:38:42.844071 16549 raft_consensus.cc:374] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:42.844182 16549 raft_consensus.cc:729] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8750beb01b443cfa153e6834e2bc4bc, State: Initialized, Role: FOLLOWER
I20260619 06:38:42.844741 16549 consensus_queue.cc:260] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 } } peers { permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 } } peers { permanent_uuid: "4fbd8ab31f454058830d99dd83520585" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44453 } }
I20260619 06:38:42.845835 16548 raft_consensus.cc:348] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 } } peers { permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 } } peers { permanent_uuid: "4fbd8ab31f454058830d99dd83520585" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44453 } }
I20260619 06:38:42.846295 16548 raft_consensus.cc:374] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:42.846403 16548 raft_consensus.cc:729] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab36a41c5c5405a95dab40b0a74ae58, State: Initialized, Role: FOLLOWER
I20260619 06:38:42.847020 16548 consensus_queue.cc:260] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 } } peers { permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 } } peers { permanent_uuid: "4fbd8ab31f454058830d99dd83520585" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44453 } }
I20260619 06:38:42.847581 16395 heartbeater.cc:503] Master 127.28.104.254:38715 was elected leader, sending a full tablet report...
I20260619 06:38:42.848207 16549 ts_tablet_manager.cc:1428] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260619 06:38:42.850910 16548 ts_tablet_manager.cc:1428] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: Time spent starting tablet: real 0.015s user 0.004s sys 0.009s
I20260619 06:38:42.873991 16553 raft_consensus.cc:482] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:38:42.874301 16553 raft_consensus.cc:504] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 } } peers { permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 } } peers { permanent_uuid: "4fbd8ab31f454058830d99dd83520585" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44453 } }
I20260619 06:38:42.876302 16553 leader_election.cc:288] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8750beb01b443cfa153e6834e2bc4bc (127.28.104.194:36163), bab36a41c5c5405a95dab40b0a74ae58 (127.28.104.193:42617)
I20260619 06:38:42.881945 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.876691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:48737 (local address 127.28.104.194:36163)
0619 06:38:42.876911 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:42.876918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:42.877174 (+ 256us) server_negotiation.cc:407] Connection header received
0619 06:38:42.877397 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:42.877403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:42.877465 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:42.877554 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:42.877923 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.878656 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.880273 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.881065 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.881214 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.881285 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:42.881559 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 06:38:42.881680 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:38:42.881961 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.876701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:36163 (local address 127.28.104.195:48737)
0619 06:38:42.877079 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:42.877099 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:42.877244 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:42.877578 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:42.877582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:42.877610 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:42.877795 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.877804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.878788 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.878795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.880130 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.880144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.881214 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.881220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.881414 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.881438 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:38:42.881534 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:38:42.881662 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":36}
I20260619 06:38:42.882740 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.876795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:59387 (local address 127.28.104.193:42617)
0619 06:38:42.877013 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:42.877049 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:42.877689 (+ 640us) server_negotiation.cc:407] Connection header received
0619 06:38:42.877883 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:42.877887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:42.877944 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:42.878036 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:42.878549 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.879219 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.880941 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.881765 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.881933 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.882005 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:42.882340 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 06:38:42.882465 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:38:42.882874 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.876761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:42617 (local address 127.28.104.195:59387)
0619 06:38:42.877581 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:42.877595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:42.877737 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:42.878046 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:42.878051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:42.878073 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:42.878388 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.878396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.879363 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.879369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.880784 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:42.880796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.881973 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.881980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:42.882165 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.882204 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:38:42.882311 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 06:38:42.882427 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":35,"thread_start_us":476,"threads_started":1}
I20260619 06:38:42.883247 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fff1643ba54a5bab97820388c89cb7" candidate_uuid: "4fbd8ab31f454058830d99dd83520585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8750beb01b443cfa153e6834e2bc4bc" is_pre_election: true
I20260619 06:38:42.883824 16349 raft_consensus.cc:2454] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fbd8ab31f454058830d99dd83520585 in term 0.
I20260619 06:38:42.883900 16213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fff1643ba54a5bab97820388c89cb7" candidate_uuid: "4fbd8ab31f454058830d99dd83520585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab36a41c5c5405a95dab40b0a74ae58" is_pre_election: true
I20260619 06:38:42.884467 16213 raft_consensus.cc:2454] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fbd8ab31f454058830d99dd83520585 in term 0.
I20260619 06:38:42.884567 16417 leader_election.cc:302] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fbd8ab31f454058830d99dd83520585, e8750beb01b443cfa153e6834e2bc4bc; no voters:
I20260619 06:38:42.885010 16553 raft_consensus.cc:2781] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:38:42.885175 16553 raft_consensus.cc:482] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:38:42.885299 16553 raft_consensus.cc:3037] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:42.887678 16553 raft_consensus.cc:504] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 } } peers { permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 } } peers { permanent_uuid: "4fbd8ab31f454058830d99dd83520585" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44453 } }
I20260619 06:38:42.888260 16553 leader_election.cc:288] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [CANDIDATE]: Term 1 election: Requested vote from peers e8750beb01b443cfa153e6834e2bc4bc (127.28.104.194:36163), bab36a41c5c5405a95dab40b0a74ae58 (127.28.104.193:42617)
I20260619 06:38:42.888602 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fff1643ba54a5bab97820388c89cb7" candidate_uuid: "4fbd8ab31f454058830d99dd83520585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8750beb01b443cfa153e6834e2bc4bc"
I20260619 06:38:42.888684 16213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fff1643ba54a5bab97820388c89cb7" candidate_uuid: "4fbd8ab31f454058830d99dd83520585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab36a41c5c5405a95dab40b0a74ae58"
I20260619 06:38:42.888813 16349 raft_consensus.cc:3037] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:42.888882 16213 raft_consensus.cc:3037] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:42.892043 16213 raft_consensus.cc:2454] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fbd8ab31f454058830d99dd83520585 in term 1.
I20260619 06:38:42.892046 16349 raft_consensus.cc:2454] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fbd8ab31f454058830d99dd83520585 in term 1.
I20260619 06:38:42.892436 16417 leader_election.cc:302] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fbd8ab31f454058830d99dd83520585, e8750beb01b443cfa153e6834e2bc4bc; no voters:
I20260619 06:38:42.892719 16553 raft_consensus.cc:2781] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:42.893198 16553 raft_consensus.cc:686] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 1 LEADER]: Becoming Leader. State: Replica: 4fbd8ab31f454058830d99dd83520585, State: Running, Role: LEADER
I20260619 06:38:42.893664 16553 consensus_queue.cc:237] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 } } peers { permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 } } peers { permanent_uuid: "4fbd8ab31f454058830d99dd83520585" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44453 } }
I20260619 06:38:42.898118 16067 catalog_manager.cc:5495] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fbd8ab31f454058830d99dd83520585 (127.28.104.195). New cstate: current_term: 1 leader_uuid: "4fbd8ab31f454058830d99dd83520585" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fbd8ab31f454058830d99dd83520585" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 44453 } health_report { overall_health: HEALTHY } } }
I20260619 06:38:42.930001 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.925119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.28.104.195:44453)
0619 06:38:42.925318 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:42.925322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:42.925449 (+ 127us) server_negotiation.cc:407] Connection header received
0619 06:38:42.925602 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:42.925606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:42.925689 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:42.925815 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:38:42.926129 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.926848 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.928145 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.928528 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.928613 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:42.929003 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:38:42.929550 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:38:42.929652 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:42.929727 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 06:38:42.929791 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:38:42.938098 16213 raft_consensus.cc:1261] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58 [term 1 FOLLOWER]: Refusing update from remote peer 4fbd8ab31f454058830d99dd83520585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:42.938098 16349 raft_consensus.cc:1261] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc [term 1 FOLLOWER]: Refusing update from remote peer 4fbd8ab31f454058830d99dd83520585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:42.939028 16557 consensus_queue.cc:1035] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8750beb01b443cfa153e6834e2bc4bc" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:42.939410 16553 consensus_queue.cc:1035] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bab36a41c5c5405a95dab40b0a74ae58" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:42.967324 16068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:42.967588 16068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:42.971442 16068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a241489871b4a138ac7fe389a64ef5c: Sending DeleteTablet for 3 replicas of tablet 92fff1643ba54a5bab97820388c89cb7
I20260619 06:38:42.972546 16329 tablet_service.cc:1514] Processing DeleteTablet for tablet 92fff1643ba54a5bab97820388c89cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:42 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:47436
I20260619 06:38:42.972702 16193 tablet_service.cc:1514] Processing DeleteTablet for tablet 92fff1643ba54a5bab97820388c89cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:42 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:42776
I20260619 06:38:42.972923 16464 tablet_service.cc:1514] Processing DeleteTablet for tablet 92fff1643ba54a5bab97820388c89cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:42 UTC) from {username='kudu', principal='kudu/127.28.104.254@KRBTEST.COM'} at 127.0.0.1:48692
I20260619 06:38:42.974965 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 16130
I20260619 06:38:42.985224 16572 tablet_replica.cc:330] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: stopping tablet replica
I20260619 06:38:42.985271 16571 tablet_replica.cc:330] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: stopping tablet replica
I20260619 06:38:42.985705 16571 raft_consensus.cc:2229] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:42.985755 16572 raft_consensus.cc:2229] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:38:42.986236 16571 raft_consensus.cc:2258] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:42.986555 16572 raft_consensus.cc:2258] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:42.988881 16572 ts_tablet_manager.cc:1905] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:42.989280 16571 ts_tablet_manager.cc:1905] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:42.992519 16570 tablet_replica.cc:330] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: stopping tablet replica
I20260619 06:38:42.992974 16570 raft_consensus.cc:2229] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:42.993547 16570 raft_consensus.cc:2258] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:42.996567 16572 ts_tablet_manager.cc:1918] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:42.996556 16571 ts_tablet_manager.cc:1918] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:42.996745 16572 log.cc:1196] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/92fff1643ba54a5bab97820388c89cb7
I20260619 06:38:42.996790 16571 log.cc:1196] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/92fff1643ba54a5bab97820388c89cb7
I20260619 06:38:42.997387 16572 ts_tablet_manager.cc:1939] T 92fff1643ba54a5bab97820388c89cb7 P 4fbd8ab31f454058830d99dd83520585: Deleting consensus metadata
I20260619 06:38:42.997555 16571 ts_tablet_manager.cc:1939] T 92fff1643ba54a5bab97820388c89cb7 P bab36a41c5c5405a95dab40b0a74ae58: Deleting consensus metadata
Jun 19 06:38:42 dist-test-slave-nf4n krb5kdc[16023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.193@KRBTEST.COM
I20260619 06:38:42.999235 16054 catalog_manager.cc:4841] TS 4fbd8ab31f454058830d99dd83520585 (127.28.104.195:44453): tablet 92fff1643ba54a5bab97820388c89cb7 (table test-table [id=ebd4e131566e40fb91081c5d2a101c6e]) successfully deleted
I20260619 06:38:42.999631 16054 catalog_manager.cc:4841] TS bab36a41c5c5405a95dab40b0a74ae58 (127.28.104.193:42617): tablet 92fff1643ba54a5bab97820388c89cb7 (table test-table [id=ebd4e131566e40fb91081c5d2a101c6e]) successfully deleted
I20260619 06:38:43.002707 16570 ts_tablet_manager.cc:1905] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:43.006583 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:42.976622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.28.104.193:42617)
0619 06:38:42.976855 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:42.976864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:42.987000 (+ 10136us) server_negotiation.cc:407] Connection header received
0619 06:38:42.987193 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:42.987198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:42.987300 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:42.987414 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:42.989151 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.990318 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.993914 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:42.994648 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:42.994760 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:43.000750 (+ 5990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:43.000762 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:43.000779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:43.000811 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:43.003671 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:43.004363 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:43.004376 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:43.004379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:43.004501 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:43.004927 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:43.004932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:43.004935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:43.005370 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:43.005716 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:43.005990 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 06:38:43.006200 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:38:43.007349 16254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:43.013846 16570 ts_tablet_manager.cc:1918] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:43.014022 16570 log.cc:1196] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/92fff1643ba54a5bab97820388c89cb7
I20260619 06:38:43.014582 16570 ts_tablet_manager.cc:1939] T 92fff1643ba54a5bab97820388c89cb7 P e8750beb01b443cfa153e6834e2bc4bc: Deleting consensus metadata
I20260619 06:38:43.016623 16053 catalog_manager.cc:4841] TS e8750beb01b443cfa153e6834e2bc4bc (127.28.104.194:36163): tablet 92fff1643ba54a5bab97820388c89cb7 (table test-table [id=ebd4e131566e40fb91081c5d2a101c6e]) successfully deleted
I20260619 06:38:43.098470 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 16130
I20260619 06:38:43.116329 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 16265
Jun 19 06:38:43 dist-test-slave-nf4n krb5kdc[16023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.194@KRBTEST.COM
I20260619 06:38:43.136891 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:43.117274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.28.104.194:36163)
0619 06:38:43.117480 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:43.117486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:43.117508 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:38:43.117658 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:43.117665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:43.117760 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:43.117876 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:43.119285 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:43.120424 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:43.122467 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:43.123118 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:43.123263 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:43.129203 (+ 5940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:43.129226 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:43.129247 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:43.129301 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:43.133416 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:43.134417 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:43.134428 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:43.134432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:43.134591 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:43.135076 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:43.135083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:43.135087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:43.135605 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:43.135922 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:43.136225 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 06:38:43.136459 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:38:43.137645 16390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:43.239033 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 16265
I20260619 06:38:43.253859 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 16400
Jun 19 06:38:43 dist-test-slave-nf4n krb5kdc[16023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.195@KRBTEST.COM
I20260619 06:38:43.273310 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:43.254717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.28.104.195:44453)
0619 06:38:43.254917 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:43.254925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:43.254963 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:38:43.255228 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:43.255232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:43.255313 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:43.255438 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:43.256891 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:43.257862 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:43.259305 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:43.259924 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:43.260088 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:43.265826 (+ 5738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:43.265839 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:43.265859 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:43.265915 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:43.270128 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:43.270930 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:43.270940 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:43.270943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:43.271141 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:43.271549 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:43.271555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:43.271558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:43.272069 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:43.272459 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:43.272627 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 06:38:43.272856 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 06:38:43.274101 16524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:43.396083 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 16400
I20260619 06:38:43.415081 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 16036
I20260619 06:38:43.416183 16098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:43.525517 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 16036
2026-06-19T06:38:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4189 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T06:38:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:43Z Disabled control of system clock
I20260619 06:38:43.601742 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:34721
--webserver_interface=127.28.104.254
--webserver_port=0
--builtin_ntp_servers=127.28.104.212:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:34721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:38:43.924443 16581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:43.924708 16581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:43.924798 16581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:43.931849 16581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:43.931951 16581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:43.931988 16581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:43.932044 16581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:43.932086 16581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:43.941210 16581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:34721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:34721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:43.942485 16581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:43.944252 16581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:43.955597 16587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:43.955705 16586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:43.955698 16590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:43.956429 16581 server_base.cc:1034] running on GCE node
I20260619 06:38:43.957252 16581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:43.958695 16581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:43.959882 16581 hybrid_clock.cc:648] HybridClock initialized: now 1781851123959826 us; error 48 us; skew 500 ppm
I20260619 06:38:43.960239 16581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:43.962476 16581 webserver.cc:460] Webserver started at http://127.28.104.254:34871/ using document root <none> and password file <none>
I20260619 06:38:43.963140 16581 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:43.963249 16581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:43.963573 16581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:43.966434 16581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "2109266fd3e14610a0455389ab63e6b6"
format_stamp: "Formatted at 2026-06-19 06:38:43 on dist-test-slave-nf4n"
server_key: "512b5c4749637eaf37dc0d5fba6f87c4"
server_key_iv: "27c1a465811e4db30ad16eee10c69fdf"
server_key_version: "encryptionkey@0"
I20260619 06:38:43.967247 16581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "2109266fd3e14610a0455389ab63e6b6"
format_stamp: "Formatted at 2026-06-19 06:38:43 on dist-test-slave-nf4n"
server_key: "512b5c4749637eaf37dc0d5fba6f87c4"
server_key_iv: "27c1a465811e4db30ad16eee10c69fdf"
server_key_version: "encryptionkey@0"
I20260619 06:38:43.972173 16581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 06:38:43.975574 16596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:43.976859 16581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:38:43.977099 16581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "2109266fd3e14610a0455389ab63e6b6"
format_stamp: "Formatted at 2026-06-19 06:38:43 on dist-test-slave-nf4n"
server_key: "512b5c4749637eaf37dc0d5fba6f87c4"
server_key_iv: "27c1a465811e4db30ad16eee10c69fdf"
server_key_version: "encryptionkey@0"
I20260619 06:38:43.977314 16581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:43.998917 16581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:43.999730 16581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:44.000015 16581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:44.017869 16581 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:34721
I20260619 06:38:44.017889 16647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:34721 every 8 connection(s)
I20260619 06:38:44.019164 16581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:44.022766 16648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:44.024389 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 16581
I20260619 06:38:44.024582 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:44.025169 29091 external_mini_cluster.cc:1428] Setting key 7b01766d634954851df627759045adee
I20260619 06:38:44.036528 16648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6: Bootstrap starting.
I20260619 06:38:44.038244 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:44.026983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.28.104.254:34721)
0619 06:38:44.027954 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:44.027965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:44.028004 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:38:44.029596 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:44.029617 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:44.030065 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:44.030377 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:44.030952 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:44.032168 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:44.034077 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:44.034713 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:44.034832 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:44.035997 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:44.036005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:44.036017 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:44.036527 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:44.036622 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:44.036649 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:44.036796 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:44.037075 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 06:38:44.037360 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":381,"threads_started":1}
I20260619 06:38:44.039762 16648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:44.040735 16648 log.cc:824] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:44.043318 16648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6: No bootstrap required, opened a new log
I20260619 06:38:44.050817 16648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2109266fd3e14610a0455389ab63e6b6" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34721 } }
I20260619 06:38:44.051188 16648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:44.051309 16648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2109266fd3e14610a0455389ab63e6b6, State: Initialized, Role: FOLLOWER
I20260619 06:38:44.051905 16648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2109266fd3e14610a0455389ab63e6b6" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34721 } }
I20260619 06:38:44.052134 16648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:44.052273 16648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:44.052446 16648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:44.054961 16648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2109266fd3e14610a0455389ab63e6b6" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34721 } }
I20260619 06:38:44.055434 16648 leader_election.cc:302] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2109266fd3e14610a0455389ab63e6b6; no voters:
I20260619 06:38:44.055961 16648 leader_election.cc:288] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:44.056105 16653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:44.056768 16653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [term 1 LEADER]: Becoming Leader. State: Replica: 2109266fd3e14610a0455389ab63e6b6, State: Running, Role: LEADER
I20260619 06:38:44.057358 16653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2109266fd3e14610a0455389ab63e6b6" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34721 } }
I20260619 06:38:44.057917 16648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:44.059983 16655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2109266fd3e14610a0455389ab63e6b6. Latest consensus state: current_term: 1 leader_uuid: "2109266fd3e14610a0455389ab63e6b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2109266fd3e14610a0455389ab63e6b6" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34721 } } }
I20260619 06:38:44.060376 16655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:44.061388 16654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2109266fd3e14610a0455389ab63e6b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2109266fd3e14610a0455389ab63e6b6" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 34721 } } }
I20260619 06:38:44.061864 16654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:44.063376 16660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:44.068804 16660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:44.077463 16660 catalog_manager.cc:1347] Generated new cluster ID: fab2684c0803450fb9ddb68f7415e975
I20260619 06:38:44.077574 16660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:44.100693 16660 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:44.102164 16660 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:44.113214 16660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6: Generated new TSK 0
I20260619 06:38:44.114007 16660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:38:44.122686 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.193:0
--local_ip_for_outbound_sockets=127.28.104.193
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34721
--builtin_ntp_servers=127.28.104.212:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:38:44.458530 16672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:44.458855 16672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:44.459005 16672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:44.466466 16672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:44.466559 16672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:44.466728 16672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.193
I20260619 06:38:44.475739 16672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.104.193
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.193
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:44.477000 16672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:44.478900 16672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:44.492033 16677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:44.491976 16680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:44.491950 16678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:44.493067 16672 server_base.cc:1034] running on GCE node
I20260619 06:38:44.493706 16672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:44.494623 16672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:44.495837 16672 hybrid_clock.cc:648] HybridClock initialized: now 1781851124495803 us; error 32 us; skew 500 ppm
I20260619 06:38:44.496201 16672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:44.498485 16672 webserver.cc:460] Webserver started at http://127.28.104.193:41421/ using document root <none> and password file <none>
I20260619 06:38:44.499197 16672 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:44.499315 16672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:44.499643 16672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:44.502624 16672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/data/instance:
uuid: "6aa7354093634e05995955a4843b47ca"
format_stamp: "Formatted at 2026-06-19 06:38:44 on dist-test-slave-nf4n"
server_key: "a29acd7f175224004adafbdbd5d395eb"
server_key_iv: "65c455f3b1b043b17da39cd241b20b56"
server_key_version: "encryptionkey@0"
I20260619 06:38:44.503453 16672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance:
uuid: "6aa7354093634e05995955a4843b47ca"
format_stamp: "Formatted at 2026-06-19 06:38:44 on dist-test-slave-nf4n"
server_key: "a29acd7f175224004adafbdbd5d395eb"
server_key_iv: "65c455f3b1b043b17da39cd241b20b56"
server_key_version: "encryptionkey@0"
I20260619 06:38:44.508423 16672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:38:44.512022 16686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:44.513502 16672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:38:44.513669 16672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/wal
uuid: "6aa7354093634e05995955a4843b47ca"
format_stamp: "Formatted at 2026-06-19 06:38:44 on dist-test-slave-nf4n"
server_key: "a29acd7f175224004adafbdbd5d395eb"
server_key_iv: "65c455f3b1b043b17da39cd241b20b56"
server_key_version: "encryptionkey@0"
I20260619 06:38:44.513895 16672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:44.527644 16672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:44.528393 16672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:44.528681 16672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:44.529675 16672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:44.531035 16672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:44.531152 16672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:44.531277 16672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:44.531363 16672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:44.563795 16672 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.193:38849
I20260619 06:38:44.563846 16799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.193:38849 every 8 connection(s)
I20260619 06:38:44.565266 16672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/data/info.pb
I20260619 06:38:44.569967 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 16672
I20260619 06:38:44.570137 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/wal/instance
I20260619 06:38:44.570556 29091 external_mini_cluster.cc:1428] Setting key 88b0e7553d780e2a60f0d1f1fff9bfc1
I20260619 06:38:44.573251 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.194:0
--local_ip_for_outbound_sockets=127.28.104.194
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34721
--builtin_ntp_servers=127.28.104.212:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:38:44.578545 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:44.567737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.193:46629 (local address 127.28.104.254:34721)
0619 06:38:44.569289 (+ 1552us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:44.569293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:44.570195 (+ 902us) server_negotiation.cc:407] Connection header received
0619 06:38:44.572230 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:44.572233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:44.572326 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:44.572449 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:44.573593 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:44.574296 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:44.575800 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:44.576222 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:44.576346 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:44.577713 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:44.577719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:44.577724 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:44.577750 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:44.577786 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:44.577798 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:44.577897 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:44.578111 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 06:38:44.578206 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1440,"thread_start_us":1336,"threads_started":1}
I20260619 06:38:44.579493 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:44.568532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34721 (local address 127.28.104.193:46629)
0619 06:38:44.570016 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:44.570081 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:44.571950 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:44.572638 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:44.572647 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:44.572734 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:44.573419 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:44.573449 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:44.574438 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:44.574444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:44.575655 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:44.575667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:44.576456 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:44.576463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:44.576677 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:44.577408 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:38:44.577411 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:44.577450 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:38:44.577524 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:44.577897 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:44.577910 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:44.577920 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 06:38:44.578086 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 06:38:44.578475 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":457,"threads_started":1}
I20260619 06:38:44.581046 16800 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34721
I20260619 06:38:44.581365 16800 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:44.582219 16800 heartbeater.cc:511] Master 127.28.104.254:34721 requested a full tablet report, sending...
I20260619 06:38:44.584226 16613 ts_manager.cc:194] Registered new tserver with Master: 6aa7354093634e05995955a4843b47ca (127.28.104.193:38849)
I20260619 06:38:44.585696 16613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.193:46629
W20260619 06:38:44.905658 16804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:44.905930 16804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:44.906102 16804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:44.913236 16804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:44.913337 16804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:44.913450 16804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.194
I20260619 06:38:44.922524 16804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.104.194
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.194
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:44.923797 16804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:44.925575 16804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:44.939030 16809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:44.939062 16812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:44.939030 16810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:44.940237 16804 server_base.cc:1034] running on GCE node
I20260619 06:38:44.940855 16804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:44.941861 16804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:44.943089 16804 hybrid_clock.cc:648] HybridClock initialized: now 1781851124943009 us; error 60 us; skew 500 ppm
I20260619 06:38:44.943406 16804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:44.945703 16804 webserver.cc:460] Webserver started at http://127.28.104.194:37505/ using document root <none> and password file <none>
I20260619 06:38:44.946404 16804 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:44.946517 16804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:44.946863 16804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:44.949669 16804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/data/instance:
uuid: "92fbaa660286433892959b92d1eedeb8"
format_stamp: "Formatted at 2026-06-19 06:38:44 on dist-test-slave-nf4n"
server_key: "9a28f92d7fa213c5cea12591d0c64257"
server_key_iv: "7faa4c19ec462219d4b273047ec4910b"
server_key_version: "encryptionkey@0"
I20260619 06:38:44.950388 16804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance:
uuid: "92fbaa660286433892959b92d1eedeb8"
format_stamp: "Formatted at 2026-06-19 06:38:44 on dist-test-slave-nf4n"
server_key: "9a28f92d7fa213c5cea12591d0c64257"
server_key_iv: "7faa4c19ec462219d4b273047ec4910b"
server_key_version: "encryptionkey@0"
I20260619 06:38:44.955400 16804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:38:44.959012 16818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:44.960311 16804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:38:44.960479 16804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/wal
uuid: "92fbaa660286433892959b92d1eedeb8"
format_stamp: "Formatted at 2026-06-19 06:38:44 on dist-test-slave-nf4n"
server_key: "9a28f92d7fa213c5cea12591d0c64257"
server_key_iv: "7faa4c19ec462219d4b273047ec4910b"
server_key_version: "encryptionkey@0"
I20260619 06:38:44.960680 16804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:44.980671 16804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:44.981541 16804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:44.981839 16804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:44.982744 16804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:44.984174 16804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:44.984279 16804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:44.984396 16804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:44.984479 16804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:45.017858 16804 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.194:37945
I20260619 06:38:45.017902 16931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.194:37945 every 8 connection(s)
I20260619 06:38:45.019192 16804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/data/info.pb
I20260619 06:38:45.028278 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 16804
I20260619 06:38:45.028470 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/wal/instance
I20260619 06:38:45.029065 29091 external_mini_cluster.cc:1428] Setting key b002d307558839efe48b0fbbfaec687d
I20260619 06:38:45.032147 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.104.195:0
--local_ip_for_outbound_sockets=127.28.104.195
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34721
--builtin_ntp_servers=127.28.104.212:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:38:45.033054 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.021834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:56333 (local address 127.28.104.254:34721)
0619 06:38:45.022002 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.022007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.023525 (+ 1518us) server_negotiation.cc:407] Connection header received
0619 06:38:45.025383 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.025389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.025472 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.025574 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:45.026541 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.027685 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.029974 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.030624 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.030742 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.032138 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:45.032144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:45.032149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:45.032172 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:45.032202 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:45.032214 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:45.032318 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.032617 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.032714 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:38:45.034132 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.022465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34721 (local address 127.28.104.194:56333)
0619 06:38:45.023336 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:45.023404 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:45.024997 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:45.025755 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:45.025762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:45.025844 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:45.026398 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.026414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.027812 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.027816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.029809 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.029822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.030831 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.030839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.031123 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.031824 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:38:45.031827 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:45.031867 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:38:45.031946 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:45.032385 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:45.032395 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:45.032402 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:38:45.032597 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 06:38:45.032996 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":264,"threads_started":1}
I20260619 06:38:45.035657 16932 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34721
I20260619 06:38:45.035959 16932 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:45.036670 16932 heartbeater.cc:511] Master 127.28.104.254:34721 requested a full tablet report, sending...
I20260619 06:38:45.038029 16613 ts_manager.cc:194] Registered new tserver with Master: 92fbaa660286433892959b92d1eedeb8 (127.28.104.194:37945)
I20260619 06:38:45.038834 16613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.194:56333
W20260619 06:38:45.363395 16935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:45.363729 16935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:45.363873 16935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:45.371081 16935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:45.371171 16935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:45.371332 16935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.104.195
I20260619 06:38:45.380314 16935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.104.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.104.195
--webserver_port=0
--tserver_master_addrs=127.28.104.254:34721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.104.195
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:45.381774 16935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:45.383512 16935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:45.396605 16943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:45.396831 16940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:45.396860 16941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:45.397975 16935 server_base.cc:1034] running on GCE node
I20260619 06:38:45.398680 16935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:45.399664 16935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:45.400888 16935 hybrid_clock.cc:648] HybridClock initialized: now 1781851125400838 us; error 46 us; skew 500 ppm
I20260619 06:38:45.401259 16935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:38:45.403425 16935 webserver.cc:460] Webserver started at http://127.28.104.195:46611/ using document root <none> and password file <none>
I20260619 06:38:45.404095 16935 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:45.404232 16935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:45.404536 16935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:45.407377 16935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/data/instance:
uuid: "ec192a81f1574a379b9c2599cd6df43a"
format_stamp: "Formatted at 2026-06-19 06:38:45 on dist-test-slave-nf4n"
server_key: "dd9eca860aaefe111d9e278a45aa9452"
server_key_iv: "42ed229878486707e6e9a6c93980d9e7"
server_key_version: "encryptionkey@0"
I20260619 06:38:45.408138 16935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance:
uuid: "ec192a81f1574a379b9c2599cd6df43a"
format_stamp: "Formatted at 2026-06-19 06:38:45 on dist-test-slave-nf4n"
server_key: "dd9eca860aaefe111d9e278a45aa9452"
server_key_iv: "42ed229878486707e6e9a6c93980d9e7"
server_key_version: "encryptionkey@0"
I20260619 06:38:45.413091 16935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:38:45.416296 16949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:45.417660 16935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:38:45.417824 16935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/wal
uuid: "ec192a81f1574a379b9c2599cd6df43a"
format_stamp: "Formatted at 2026-06-19 06:38:45 on dist-test-slave-nf4n"
server_key: "dd9eca860aaefe111d9e278a45aa9452"
server_key_iv: "42ed229878486707e6e9a6c93980d9e7"
server_key_version: "encryptionkey@0"
I20260619 06:38:45.418066 16935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:45.432284 16935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:45.433144 16935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:45.433459 16935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:45.434412 16935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:38:45.435810 16935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:38:45.435928 16935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:45.436055 16935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:38:45.436141 16935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:45.468616 16935 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.195:45095
I20260619 06:38:45.468679 17061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.195:45095 every 8 connection(s)
I20260619 06:38:45.469992 16935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/data/info.pb
I20260619 06:38:45.478276 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 16935
I20260619 06:38:45.478472 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/wal/instance
I20260619 06:38:45.479029 29091 external_mini_cluster.cc:1428] Setting key f7b4e0ac2084d43b37b40da06f80be78
I20260619 06:38:45.483153 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.472537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.195:34045 (local address 127.28.104.254:34721)
0619 06:38:45.472711 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.472716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.474392 (+ 1676us) server_negotiation.cc:407] Connection header received
0619 06:38:45.476103 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.476107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.476175 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.476264 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:45.477447 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.478167 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.480301 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.480880 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.481015 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.482350 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:45.482361 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:45.482365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:45.482381 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:45.482410 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:45.482422 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:45.482511 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.482719 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.482811 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:38:45.484068 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.473162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:34721 (local address 127.28.104.195:34045)
0619 06:38:45.474233 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:45.474277 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:45.475859 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:45.476451 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:45.476458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:45.476539 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:38:45.477241 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.477260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.478329 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.478335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.480120 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.480135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.481126 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.481133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.481353 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.482061 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:38:45.482064 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:38:45.482100 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:38:45.482169 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:38:45.482510 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:38:45.482521 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:38:45.482531 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 06:38:45.482702 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 06:38:45.483062 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":341,"threads_started":1}
I20260619 06:38:45.485347 17062 heartbeater.cc:347] Connected to a master server at 127.28.104.254:34721
I20260619 06:38:45.485594 17062 heartbeater.cc:464] Registering TS with master...
I20260619 06:38:45.486274 17062 heartbeater.cc:511] Master 127.28.104.254:34721 requested a full tablet report, sending...
I20260619 06:38:45.487569 16613 ts_manager.cc:194] Registered new tserver with Master: ec192a81f1574a379b9c2599cd6df43a (127.28.104.195:45095)
I20260619 06:38:45.488338 16613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.104.195:34045
I20260619 06:38:45.494365 29091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:38:45.504299 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.498415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.28.104.254:34721)
0619 06:38:45.498597 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.498606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.498962 (+ 356us) server_negotiation.cc:407] Connection header received
0619 06:38:45.499158 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.499164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.499253 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.499354 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:45.499860 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.500557 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.502038 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.502455 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.502601 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.503582 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:45.503588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:45.503593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:45.503617 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:45.503648 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:45.503660 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:45.503747 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.503912 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.504005 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:38:45.507813 16613 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:38:45.514451 16613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:38:45.539176 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.525840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.28.104.193:38849)
0619 06:38:45.526423 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.526432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.527338 (+ 906us) server_negotiation.cc:407] Connection header received
0619 06:38:45.527764 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.527794 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.528091 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.528282 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:45.529455 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.531226 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.534571 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.536548 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.536734 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.537637 (+ 903us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.538412 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.538658 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":1022,"threads_started":1}
I20260619 06:38:45.540364 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.525624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:38849 (local address 127.0.0.1:60564)
0619 06:38:45.527193 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:45.527232 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:45.527491 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:45.528656 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:45.528664 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:45.528704 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:45.529211 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.529228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.531452 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.531461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.534283 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.534303 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.537820 (+ 3517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.537826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.538058 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.538141 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 06:38:45.539741 (+ 1600us) client_negotiation.cc:240] Negotiation successful
0619 06:38:45.539987 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"threads_started":1}
I20260619 06:38:45.545815 16735 tablet_service.cc:1467] Processing CreateTablet for tablet c8633155629b4360a73992e648099c25 (DEFAULT_TABLE table=test-table [id=ad9601ae6886499897d88e642fb1ef01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:45.547235 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8633155629b4360a73992e648099c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:45.549263 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.526003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.194:37945 (local address 127.0.0.1:36048)
0619 06:38:45.540855 (+ 14852us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:45.540885 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:45.541018 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:45.542232 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:45.542236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:45.542262 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:45.542431 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.542441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.544452 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.544457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.546317 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.546331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.548454 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.548460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.548633 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.548670 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:38:45.548774 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:38:45.548905 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14666,"thread_start_us":763,"threads_started":1}
I20260619 06:38:45.549360 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.526707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.28.104.194:37945)
0619 06:38:45.541565 (+ 14858us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.541577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.541612 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:38:45.541722 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.541731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.542025 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.542613 (+ 588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:45.542837 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.544312 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.546483 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.547970 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.548134 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.548364 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.548861 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.549057 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14698,"thread_start_us":14595,"threads_started":1}
I20260619 06:38:45.550674 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.527705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:45095 (local address 127.0.0.1:38236)
0619 06:38:45.541115 (+ 13410us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:45.541137 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:45.541266 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:45.543775 (+ 2509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:45.543780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:45.543804 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:45.544162 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.544185 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.545560 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.545568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.547368 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.547383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.549175 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.549184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.550083 (+ 899us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.550114 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:38:45.550207 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:38:45.550355 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13219,"thread_start_us":13939,"threads_started":1}
I20260619 06:38:45.551465 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.528362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.28.104.195:45095)
0619 06:38:45.543206 (+ 14844us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.543214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.543246 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:38:45.543347 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.543356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.543570 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.543755 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:45.544367 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.545438 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.547563 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.548933 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.549745 (+ 812us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.549965 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.550903 (+ 938us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.551075 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14671,"thread_start_us":14506,"threads_started":1}
I20260619 06:38:45.552577 16866 tablet_service.cc:1467] Processing CreateTablet for tablet c8633155629b4360a73992e648099c25 (DEFAULT_TABLE table=test-table [id=ad9601ae6886499897d88e642fb1ef01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:45.554066 16866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8633155629b4360a73992e648099c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:45.555205 16997 tablet_service.cc:1467] Processing CreateTablet for tablet c8633155629b4360a73992e648099c25 (DEFAULT_TABLE table=test-table [id=ad9601ae6886499897d88e642fb1ef01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:38:45.556648 16997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8633155629b4360a73992e648099c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:45.560302 17080 tablet_bootstrap.cc:492] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: Bootstrap starting.
I20260619 06:38:45.564029 17080 tablet_bootstrap.cc:654] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:45.565621 17080 log.cc:824] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:45.575006 17082 tablet_bootstrap.cc:492] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: Bootstrap starting.
I20260619 06:38:45.576570 17083 tablet_bootstrap.cc:492] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: Bootstrap starting.
I20260619 06:38:45.579636 17082 tablet_bootstrap.cc:654] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:45.580127 17083 tablet_bootstrap.cc:654] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:45.581132 17082 log.cc:824] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:45.581346 17083 log.cc:824] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:45.587934 17083 tablet_bootstrap.cc:492] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: No bootstrap required, opened a new log
I20260619 06:38:45.587939 17080 tablet_bootstrap.cc:492] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: No bootstrap required, opened a new log
I20260619 06:38:45.588289 17083 ts_tablet_manager.cc:1397] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260619 06:38:45.588650 17082 tablet_bootstrap.cc:492] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: No bootstrap required, opened a new log
I20260619 06:38:45.588449 17080 ts_tablet_manager.cc:1397] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.015s
I20260619 06:38:45.588943 17082 ts_tablet_manager.cc:1397] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260619 06:38:45.589268 16800 heartbeater.cc:503] Master 127.28.104.254:34721 was elected leader, sending a full tablet report...
I20260619 06:38:45.597852 17083 raft_consensus.cc:348] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 } } peers { permanent_uuid: "92fbaa660286433892959b92d1eedeb8" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 37945 } } peers { permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 } }
I20260619 06:38:45.598253 17083 raft_consensus.cc:374] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:45.598359 17083 raft_consensus.cc:729] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec192a81f1574a379b9c2599cd6df43a, State: Initialized, Role: FOLLOWER
I20260619 06:38:45.598987 17083 consensus_queue.cc:260] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 } } peers { permanent_uuid: "92fbaa660286433892959b92d1eedeb8" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 37945 } } peers { permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 } }
I20260619 06:38:45.599304 17082 raft_consensus.cc:348] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 } } peers { permanent_uuid: "92fbaa660286433892959b92d1eedeb8" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 37945 } } peers { permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 } }
I20260619 06:38:45.599764 17082 raft_consensus.cc:374] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:45.599861 17082 raft_consensus.cc:729] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fbaa660286433892959b92d1eedeb8, State: Initialized, Role: FOLLOWER
I20260619 06:38:45.600272 17062 heartbeater.cc:503] Master 127.28.104.254:34721 was elected leader, sending a full tablet report...
I20260619 06:38:45.600544 17082 consensus_queue.cc:260] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 } } peers { permanent_uuid: "92fbaa660286433892959b92d1eedeb8" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 37945 } } peers { permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 } }
I20260619 06:38:45.600826 17083 ts_tablet_manager.cc:1428] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 06:38:45.601902 16932 heartbeater.cc:503] Master 127.28.104.254:34721 was elected leader, sending a full tablet report...
I20260619 06:38:45.601639 17080 raft_consensus.cc:348] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 } } peers { permanent_uuid: "92fbaa660286433892959b92d1eedeb8" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 37945 } } peers { permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 } }
I20260619 06:38:45.602089 17080 raft_consensus.cc:374] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:45.602181 17080 raft_consensus.cc:729] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa7354093634e05995955a4843b47ca, State: Initialized, Role: FOLLOWER
I20260619 06:38:45.602923 17082 ts_tablet_manager.cc:1428] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260619 06:38:45.602746 17080 consensus_queue.cc:260] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 } } peers { permanent_uuid: "92fbaa660286433892959b92d1eedeb8" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 37945 } } peers { permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 } }
I20260619 06:38:45.604499 17080 ts_tablet_manager.cc:1428] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s
I20260619 06:38:45.671916 17087 raft_consensus.cc:482] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:38:45.672245 17087 raft_consensus.cc:504] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 } } peers { permanent_uuid: "92fbaa660286433892959b92d1eedeb8" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 37945 } } peers { permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 } }
I20260619 06:38:45.674027 17087 leader_election.cc:288] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aa7354093634e05995955a4843b47ca (127.28.104.193:38849), ec192a81f1574a379b9c2599cd6df43a (127.28.104.195:45095)
I20260619 06:38:45.682865 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.674601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:43715 (local address 127.28.104.195:45095)
0619 06:38:45.674824 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.674832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.675427 (+ 595us) server_negotiation.cc:407] Connection header received
0619 06:38:45.675581 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.675586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.675659 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.675754 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:45.676502 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.677628 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.680751 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.681609 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.681706 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.681757 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.682472 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.682608 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:38:45.682865 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.674628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.104.194:51125 (local address 127.28.104.193:38849)
0619 06:38:45.674886 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.674892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.675336 (+ 444us) server_negotiation.cc:407] Connection header received
0619 06:38:45.675555 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.675560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.675651 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.675754 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:38:45.676502 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.677620 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.680733 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.681757 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.681907 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.681960 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.682504 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.682624 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:38:45.683104 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.674319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.193:38849 (local address 127.28.104.194:51125)
0619 06:38:45.675215 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:45.675242 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:45.675407 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:45.675788 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:45.675793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:45.675879 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:45.676362 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.676370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.677777 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.677784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.680502 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.680518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.681996 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.682002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.682257 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.682314 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 06:38:45.682443 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 06:38:45.682542 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":212,"spinlock_wait_cycles":23808,"thread_start_us":493,"threads_started":1}
I20260619 06:38:45.683132 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.674430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.195:45095 (local address 127.28.104.194:43715)
0619 06:38:45.675336 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0619 06:38:45.675351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:38:45.675443 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:38:45.675788 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:38:45.675793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:38:45.675819 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:38:45.676352 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.676365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.677747 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.677753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.680505 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:38:45.680518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.681973 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.681984 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:38:45.682260 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.682310 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 06:38:45.682408 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 06:38:45.682542 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":244,"thread_start_us":558,"threads_started":1}
I20260619 06:38:45.684230 16755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8633155629b4360a73992e648099c25" candidate_uuid: "92fbaa660286433892959b92d1eedeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa7354093634e05995955a4843b47ca" is_pre_election: true
I20260619 06:38:45.684758 16755 raft_consensus.cc:2454] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92fbaa660286433892959b92d1eedeb8 in term 0.
I20260619 06:38:45.685400 16819 leader_election.cc:302] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aa7354093634e05995955a4843b47ca, 92fbaa660286433892959b92d1eedeb8; no voters:
I20260619 06:38:45.685519 17017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8633155629b4360a73992e648099c25" candidate_uuid: "92fbaa660286433892959b92d1eedeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec192a81f1574a379b9c2599cd6df43a" is_pre_election: true
I20260619 06:38:45.685763 17087 raft_consensus.cc:2781] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:38:45.685901 17087 raft_consensus.cc:482] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:38:45.685999 17087 raft_consensus.cc:3037] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:45.686097 17017 raft_consensus.cc:2454] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92fbaa660286433892959b92d1eedeb8 in term 0.
I20260619 06:38:45.688248 17087 raft_consensus.cc:504] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 } } peers { permanent_uuid: "92fbaa660286433892959b92d1eedeb8" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 37945 } } peers { permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 } }
I20260619 06:38:45.688750 17087 leader_election.cc:288] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [CANDIDATE]: Term 1 election: Requested vote from peers 6aa7354093634e05995955a4843b47ca (127.28.104.193:38849), ec192a81f1574a379b9c2599cd6df43a (127.28.104.195:45095)
I20260619 06:38:45.689236 17017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8633155629b4360a73992e648099c25" candidate_uuid: "92fbaa660286433892959b92d1eedeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec192a81f1574a379b9c2599cd6df43a"
I20260619 06:38:45.689255 16755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8633155629b4360a73992e648099c25" candidate_uuid: "92fbaa660286433892959b92d1eedeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa7354093634e05995955a4843b47ca"
I20260619 06:38:45.689455 17017 raft_consensus.cc:3037] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:45.689455 16755 raft_consensus.cc:3037] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:45.692639 16755 raft_consensus.cc:2454] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92fbaa660286433892959b92d1eedeb8 in term 1.
I20260619 06:38:45.692639 17017 raft_consensus.cc:2454] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92fbaa660286433892959b92d1eedeb8 in term 1.
I20260619 06:38:45.693068 16822 leader_election.cc:302] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92fbaa660286433892959b92d1eedeb8, ec192a81f1574a379b9c2599cd6df43a; no voters:
I20260619 06:38:45.693387 17087 raft_consensus.cc:2781] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:45.693822 17087 raft_consensus.cc:686] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 1 LEADER]: Becoming Leader. State: Replica: 92fbaa660286433892959b92d1eedeb8, State: Running, Role: LEADER
I20260619 06:38:45.694264 17087 consensus_queue.cc:237] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 } } peers { permanent_uuid: "92fbaa660286433892959b92d1eedeb8" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 37945 } } peers { permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 } }
I20260619 06:38:45.698684 16611 catalog_manager.cc:5495] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92fbaa660286433892959b92d1eedeb8 (127.28.104.194). New cstate: current_term: 1 leader_uuid: "92fbaa660286433892959b92d1eedeb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92fbaa660286433892959b92d1eedeb8" member_type: VOTER last_known_addr { host: "127.28.104.194" port: 37945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 } health_report { overall_health: UNKNOWN } } }
W20260619 06:38:45.721666 17063 tablet.cc:2369] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:38:45.728217 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.722936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.28.104.194:37945)
0619 06:38:45.723169 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.723175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.723310 (+ 135us) server_negotiation.cc:407] Connection header received
0619 06:38:45.723547 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.723550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.723646 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.723804 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:38:45.724137 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.724839 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.726173 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.726593 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.726724 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.727197 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:38:45.727664 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:38:45.727828 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.727880 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.727959 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:38:45.736881 17017 raft_consensus.cc:1261] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a [term 1 FOLLOWER]: Refusing update from remote peer 92fbaa660286433892959b92d1eedeb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:45.736881 16755 raft_consensus.cc:1261] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca [term 1 FOLLOWER]: Refusing update from remote peer 92fbaa660286433892959b92d1eedeb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:38:45.737998 17091 consensus_queue.cc:1035] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aa7354093634e05995955a4843b47ca" member_type: VOTER last_known_addr { host: "127.28.104.193" port: 38849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:45.738354 17087 consensus_queue.cc:1035] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec192a81f1574a379b9c2599cd6df43a" member_type: VOTER last_known_addr { host: "127.28.104.195" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:38:45.761631 16611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:45.761885 16611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:38:45.765617 16611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2109266fd3e14610a0455389ab63e6b6: Sending DeleteTablet for 3 replicas of tablet c8633155629b4360a73992e648099c25
I20260619 06:38:45.766721 16735 tablet_service.cc:1514] Processing DeleteTablet for tablet c8633155629b4360a73992e648099c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:45 UTC) from {username='slave'} at 127.0.0.1:60564
I20260619 06:38:45.766906 16997 tablet_service.cc:1514] Processing DeleteTablet for tablet c8633155629b4360a73992e648099c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:45 UTC) from {username='slave'} at 127.0.0.1:38236
I20260619 06:38:45.767099 16866 tablet_service.cc:1514] Processing DeleteTablet for tablet c8633155629b4360a73992e648099c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:38:45 UTC) from {username='slave'} at 127.0.0.1:36048
I20260619 06:38:45.769280 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 16672
I20260619 06:38:45.779125 17104 tablet_replica.cc:330] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: stopping tablet replica
I20260619 06:38:45.779796 17104 raft_consensus.cc:2229] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:45.780553 17104 raft_consensus.cc:2258] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:45.782986 17104 ts_tablet_manager.cc:1905] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:45.792032 17104 ts_tablet_manager.cc:1918] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:45.792282 17104 log.cc:1196] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-0/wal/wals/c8633155629b4360a73992e648099c25
I20260619 06:38:45.792965 17104 ts_tablet_manager.cc:1939] T c8633155629b4360a73992e648099c25 P 6aa7354093634e05995955a4843b47ca: Deleting consensus metadata
I20260619 06:38:45.795104 16597 catalog_manager.cc:4841] TS 6aa7354093634e05995955a4843b47ca (127.28.104.193:38849): tablet c8633155629b4360a73992e648099c25 (table test-table [id=ad9601ae6886499897d88e642fb1ef01]) successfully deleted
I20260619 06:38:45.796486 17106 tablet_replica.cc:330] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: stopping tablet replica
I20260619 06:38:45.796912 17106 raft_consensus.cc:2229] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:38:45.797627 17106 raft_consensus.cc:2258] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:45.799901 17106 ts_tablet_manager.cc:1905] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:45.803017 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.770722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.28.104.193:38849)
0619 06:38:45.770935 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.770944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.794768 (+ 23824us) server_negotiation.cc:407] Connection header received
0619 06:38:45.795253 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.795258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.795370 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.795448 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:45.795983 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.797188 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.799157 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.799739 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.799850 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.801358 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:45.801369 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:45.801387 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:45.802130 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:45.802190 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:45.802218 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:45.802298 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.802536 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.802647 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:38:45.804282 17105 tablet_replica.cc:330] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: stopping tablet replica
I20260619 06:38:45.804309 16795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:45.804790 17105 raft_consensus.cc:2229] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:38:45.805616 17105 raft_consensus.cc:2258] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:38:45.808508 17105 ts_tablet_manager.cc:1905] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:38:45.812944 17106 ts_tablet_manager.cc:1918] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:45.813174 17106 log.cc:1196] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-1/wal/wals/c8633155629b4360a73992e648099c25
I20260619 06:38:45.813867 17106 ts_tablet_manager.cc:1939] T c8633155629b4360a73992e648099c25 P 92fbaa660286433892959b92d1eedeb8: Deleting consensus metadata
I20260619 06:38:45.815299 17105 ts_tablet_manager.cc:1918] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:38:45.815451 17105 log.cc:1196] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: Deleting WAL directory at /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850805520649-29091-0/minicluster-data/ts-2/wal/wals/c8633155629b4360a73992e648099c25
I20260619 06:38:45.815994 17105 ts_tablet_manager.cc:1939] T c8633155629b4360a73992e648099c25 P ec192a81f1574a379b9c2599cd6df43a: Deleting consensus metadata
I20260619 06:38:45.816082 16600 catalog_manager.cc:4841] TS 92fbaa660286433892959b92d1eedeb8 (127.28.104.194:37945): tablet c8633155629b4360a73992e648099c25 (table test-table [id=ad9601ae6886499897d88e642fb1ef01]) successfully deleted
I20260619 06:38:45.817785 16600 catalog_manager.cc:4841] TS ec192a81f1574a379b9c2599cd6df43a (127.28.104.195:45095): tablet c8633155629b4360a73992e648099c25 (table test-table [id=ad9601ae6886499897d88e642fb1ef01]) successfully deleted
I20260619 06:38:45.918031 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 16672
I20260619 06:38:45.943001 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 16804
I20260619 06:38:45.950111 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:45.943968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.28.104.194:37945)
0619 06:38:45.944149 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:45.944156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:45.944178 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:38:45.944249 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:45.944254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:45.944338 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:45.944462 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:45.944841 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.945712 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.946984 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:45.947405 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:45.947559 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:45.948625 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:45.948634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:45.948648 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:45.949343 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:45.949391 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:45.949418 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:45.949523 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:45.949672 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 06:38:45.949774 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:38:45.950901 16927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:46.062899 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 16804
I20260619 06:38:46.082432 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 16935
I20260619 06:38:46.092904 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:46.083166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.28.104.195:45095)
0619 06:38:46.083330 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:46.083334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:46.083409 (+ 75us) server_negotiation.cc:407] Connection header received
0619 06:38:46.083551 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:46.083555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:46.083687 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:46.083775 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:46.084126 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:46.084858 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:46.086123 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:46.086497 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:46.086590 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:46.091695 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:46.091705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:46.091719 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:46.092279 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:46.092329 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:46.092353 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:46.092456 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:46.092567 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 06:38:46.092655 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:38:46.093555 17057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:46.182581 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 16935
I20260619 06:38:46.199445 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 16581
I20260619 06:38:46.200469 16643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:46.309506 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 16581
2026-06-19T06:38:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2784 ms)
[----------] 6 tests from EncryptionPolicyTest (21342 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:38:46 dist-test-slave-nf4n krb5kdc[17115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:38:46 dist-test-slave-nf4n krb5kdc[17115](info): set up 2 sockets
Jun 19 06:38:46 dist-test-slave-nf4n krb5kdc[17115](info): commencing operation
krb5kdc: starting...
W20260619 06:38:47.461355 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:38:47 dist-test-slave-nf4n krb5kdc[17115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:38:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:38:47.704631 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:48.033972 17128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:48.034323 17128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:48.034503 17128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:48.042287 17128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:48.042433 17128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:48.042523 17128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:48.042644 17128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:48.042729 17128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:48.053555 17128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:48.054867 17128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:48.056185 17128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:48.069159 17133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:48.069406 17137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:48.069487 17134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:48.070341 17128 server_base.cc:1034] running on GCE node
I20260619 06:38:48.071396 17128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:48.072847 17128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:48.074080 17128 hybrid_clock.cc:648] HybridClock initialized: now 1781851128074035 us; error 65 us; skew 500 ppm
I20260619 06:38:48.074448 17128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:48 dist-test-slave-nf4n krb5kdc[17115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:48.077790 17128 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:38:48.079264 17128 webserver.cc:460] Webserver started at http://127.0.0.1:43923/ using document root <none> and password file <none>
I20260619 06:38:48.079977 17128 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:48.080125 17128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:48.080449 17128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:48.083609 17128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "5e61247e7bec491ba5d4d9c467ebc0bb"
format_stamp: "Formatted at 2026-06-19 06:38:48 on dist-test-slave-nf4n"
server_key: "161d71bca06b58b5d7e56f96b825e27a"
server_key_iv: "689181c883ff1989774db6d6d45805a4"
server_key_version: "encryptionkey@0"
I20260619 06:38:48.084393 17128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "5e61247e7bec491ba5d4d9c467ebc0bb"
format_stamp: "Formatted at 2026-06-19 06:38:48 on dist-test-slave-nf4n"
server_key: "161d71bca06b58b5d7e56f96b825e27a"
server_key_iv: "689181c883ff1989774db6d6d45805a4"
server_key_version: "encryptionkey@0"
I20260619 06:38:48.089861 17128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 06:38:48.093232 17144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:48.094642 17128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:38:48.094830 17128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "5e61247e7bec491ba5d4d9c467ebc0bb"
format_stamp: "Formatted at 2026-06-19 06:38:48 on dist-test-slave-nf4n"
server_key: "161d71bca06b58b5d7e56f96b825e27a"
server_key_iv: "689181c883ff1989774db6d6d45805a4"
server_key_version: "encryptionkey@0"
I20260619 06:38:48.095026 17128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:48.109292 17128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:48.113430 17128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:48.113739 17128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:48.133755 17128 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43205
I20260619 06:38:48.133792 17195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43205 every 8 connection(s)
I20260619 06:38:48.135200 17128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:48.137187 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 17128
I20260619 06:38:48.137391 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:48.138211 29091 external_mini_cluster.cc:1428] Setting key 3c375b968a41729ffdcf45bc920fc850
I20260619 06:38:48.139724 17196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:38:48 dist-test-slave-nf4n krb5kdc[17115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:38:48.153750 17196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb: Bootstrap starting.
I20260619 06:38:48.158005 17196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:48.159184 17196 log.cc:824] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:48.162463 17196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb: No bootstrap required, opened a new log
I20260619 06:38:48.162591 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:48.140143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35383 (local address 127.0.0.1:43205)
0619 06:38:48.140941 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:48.140952 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:48.140991 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:38:48.142622 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:48.142643 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:48.143026 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:48.143361 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:48.145180 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:48.146568 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:48.148343 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:48.148982 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:48.149189 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:48.153957 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:48.153968 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:48.153985 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:48.154048 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:48.157950 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:48.158772 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:48.158780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:48.158788 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:48.159002 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:48.159479 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:48.159484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:48.159487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:48.160142 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:48.160384 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:48.160980 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 06:38:48.161328 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":365,"threads_started":1}
I20260619 06:38:48.166446 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:38:48.170410 17196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e61247e7bec491ba5d4d9c467ebc0bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43205 } }
I20260619 06:38:48.170748 17196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:48.170864 17196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e61247e7bec491ba5d4d9c467ebc0bb, State: Initialized, Role: FOLLOWER
I20260619 06:38:48.171547 17196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e61247e7bec491ba5d4d9c467ebc0bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43205 } }
I20260619 06:38:48.171785 17196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:48.171927 17196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:48.172129 17196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:48.174351 17196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e61247e7bec491ba5d4d9c467ebc0bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43205 } }
I20260619 06:38:48.174847 17196 leader_election.cc:302] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e61247e7bec491ba5d4d9c467ebc0bb; no voters:
I20260619 06:38:48.175488 17196 leader_election.cc:288] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:48.175598 17206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:48.176961 17206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [term 1 LEADER]: Becoming Leader. State: Replica: 5e61247e7bec491ba5d4d9c467ebc0bb, State: Running, Role: LEADER
I20260619 06:38:48.177604 17206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e61247e7bec491ba5d4d9c467ebc0bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43205 } }
I20260619 06:38:48.177912 17196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:48.180534 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e61247e7bec491ba5d4d9c467ebc0bb. Latest consensus state: current_term: 1 leader_uuid: "5e61247e7bec491ba5d4d9c467ebc0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e61247e7bec491ba5d4d9c467ebc0bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43205 } } }
I20260619 06:38:48.181087 17207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e61247e7bec491ba5d4d9c467ebc0bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e61247e7bec491ba5d4d9c467ebc0bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43205 } } }
I20260619 06:38:48.181365 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:48.181382 17207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:48.182559 17212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:48.188787 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:48.171291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51315 (local address 127.0.0.1:43205)
0619 06:38:48.171457 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:48.171463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:48.171871 (+ 408us) server_negotiation.cc:407] Connection header received
0619 06:38:48.172058 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:48.172063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:48.172139 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:48.172236 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:48.173700 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:48.174443 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:48.175723 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:48.176470 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:48.176586 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:48.179292 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:48.179297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:48.179304 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:48.179341 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:48.183720 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:48.184824 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:48.184833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:48.184839 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:48.185045 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:48.185765 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:48.185773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:48.185778 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:48.186271 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:48.186635 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:48.187944 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0619 06:38:48.188234 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:38:48.196700 17212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:48.206087 17212 catalog_manager.cc:1347] Generated new cluster ID: d24ad2978f83401cbaa1dd4c3e5960e6
I20260619 06:38:48.206223 17212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:48.224727 17212 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:48.226538 17212 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:48.238821 17212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e61247e7bec491ba5d4d9c467ebc0bb: Generated new TSK 0
I20260619 06:38:48.239856 17212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:38:48.251279 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17128
I20260619 06:38:48.253351 17191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:48.346051 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 17128
W20260619 06:38:48.363682 17113 connection.cc:535] client connection to 127.0.0.1:43205 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43205): Connection reset by peer (error 104)
2026-06-19T06:38:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2030 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:38:48 dist-test-slave-nf4n krb5kdc[17229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:38:48 dist-test-slave-nf4n krb5kdc[17229](info): set up 2 sockets
Jun 19 06:38:48 dist-test-slave-nf4n krb5kdc[17229](info): commencing operation
krb5kdc: starting...
W20260619 06:38:49.457813 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:38:49 dist-test-slave-nf4n krb5kdc[17229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:38:49.701566 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:50.028187 17242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:50.028492 17242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:50.028589 17242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:50.035892 17242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:50.036006 17242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:50.036054 17242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:50.036116 17242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:50.036157 17242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:50.046066 17242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:50.047344 17242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:50.048604 17242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:50.060622 17250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:50.060736 17247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:50.060709 17248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:50.061627 17242 server_base.cc:1034] running on GCE node
I20260619 06:38:50.062427 17242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:50.063714 17242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:50.064915 17242 hybrid_clock.cc:648] HybridClock initialized: now 1781851130064873 us; error 33 us; skew 500 ppm
I20260619 06:38:50.065274 17242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:50 dist-test-slave-nf4n krb5kdc[17229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:50.068326 17242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:38:50.069841 17242 webserver.cc:460] Webserver started at http://127.0.0.1:46647/ using document root <none> and password file <none>
I20260619 06:38:50.070483 17242 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:50.070612 17242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:50.070921 17242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:50.073873 17242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "ac25513dc1374d469b492f99a47e98d9"
format_stamp: "Formatted at 2026-06-19 06:38:50 on dist-test-slave-nf4n"
server_key: "a45180f730730b0b36ab8c2714ad8159"
server_key_iv: "43194cdc3668d0fe6afc5a7214b169a2"
server_key_version: "encryptionkey@0"
I20260619 06:38:50.074625 17242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "ac25513dc1374d469b492f99a47e98d9"
format_stamp: "Formatted at 2026-06-19 06:38:50 on dist-test-slave-nf4n"
server_key: "a45180f730730b0b36ab8c2714ad8159"
server_key_iv: "43194cdc3668d0fe6afc5a7214b169a2"
server_key_version: "encryptionkey@0"
I20260619 06:38:50.079733 17242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:38:50.083195 17257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:50.084555 17242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:38:50.084692 17242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "ac25513dc1374d469b492f99a47e98d9"
format_stamp: "Formatted at 2026-06-19 06:38:50 on dist-test-slave-nf4n"
server_key: "a45180f730730b0b36ab8c2714ad8159"
server_key_iv: "43194cdc3668d0fe6afc5a7214b169a2"
server_key_version: "encryptionkey@0"
I20260619 06:38:50.084856 17242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:50.099304 17242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:50.103084 17242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:50.103318 17242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:50.121296 17242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39201
I20260619 06:38:50.121328 17309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39201 every 8 connection(s)
I20260619 06:38:50.122634 17242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:50.123606 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 17242
I20260619 06:38:50.123769 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:50.124243 29091 external_mini_cluster.cc:1428] Setting key 8e7baadd1a5921211c81a60d3e87ab73
I20260619 06:38:50.126804 17310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:50.137545 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:50.126004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42947 (local address 127.0.0.1:39201)
0619 06:38:50.126806 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:50.126816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:50.126880 (+ 64us) server_negotiation.cc:407] Connection header received
0619 06:38:50.128476 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:50.128496 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:50.128876 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:50.129272 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:50.129832 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:50.131209 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:50.133115 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:50.133757 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:50.133885 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:50.135212 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:50.135220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:50.135233 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:50.135781 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:50.135855 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:50.135880 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:50.135991 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:50.136319 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 06:38:50.136509 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":372,"threads_started":1}
I20260619 06:38:50.138921 17310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9: Bootstrap starting.
I20260619 06:38:50.141225 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:38:50.142488 17310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:50.143703 17310 log.cc:824] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:50.146549 17310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9: No bootstrap required, opened a new log
I20260619 06:38:50.153790 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:50.146820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54577 (local address 127.0.0.1:39201)
0619 06:38:50.147039 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:50.147045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:50.147422 (+ 377us) server_negotiation.cc:407] Connection header received
0619 06:38:50.147614 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:50.147618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:50.147674 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:50.147752 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:50.148352 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:50.149268 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:50.151198 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:50.151617 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:50.151792 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:50.152948 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:50.152953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:50.152956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:50.152975 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:50.153071 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:50.153098 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:50.153186 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:50.153397 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 06:38:50.153439 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:38:50.156188 17310 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac25513dc1374d469b492f99a47e98d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260619 06:38:50.156561 17310 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:50.156679 17310 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac25513dc1374d469b492f99a47e98d9, State: Initialized, Role: FOLLOWER
I20260619 06:38:50.157326 17310 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac25513dc1374d469b492f99a47e98d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260619 06:38:50.157569 17310 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:50.157723 17310 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:50.157888 17310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:50.160018 17310 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac25513dc1374d469b492f99a47e98d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260619 06:38:50.160580 17310 leader_election.cc:302] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac25513dc1374d469b492f99a47e98d9; no voters:
I20260619 06:38:50.161197 17310 leader_election.cc:288] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:50.161294 17320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:50.162047 17320 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [term 1 LEADER]: Becoming Leader. State: Replica: ac25513dc1374d469b492f99a47e98d9, State: Running, Role: LEADER
I20260619 06:38:50.162634 17320 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac25513dc1374d469b492f99a47e98d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260619 06:38:50.163266 17310 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:50.165513 17322 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac25513dc1374d469b492f99a47e98d9. Latest consensus state: current_term: 1 leader_uuid: "ac25513dc1374d469b492f99a47e98d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac25513dc1374d469b492f99a47e98d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } } }
I20260619 06:38:50.165954 17322 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:50.167656 17321 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac25513dc1374d469b492f99a47e98d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac25513dc1374d469b492f99a47e98d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } } }
I20260619 06:38:50.167939 17321 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:50.168459 17328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:50.173192 17328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:50.182018 17328 catalog_manager.cc:1347] Generated new cluster ID: 3f41c85551d5449e9c68a2ab78dab336
I20260619 06:38:50.182127 17328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:50.210227 17328 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:50.212409 17328 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:50.224241 17328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac25513dc1374d469b492f99a47e98d9: Generated new TSK 0
I20260619 06:38:50.225106 17328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:38:50.241462 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17242
W20260619 06:38:50.242671 17304 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42947
W20260619 06:38:50.243471 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17242: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:38:50.243675 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 17242
W20260619 06:38:50.260753 17227 connection.cc:535] client connection to 127.0.0.1:39201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39201): Connection reset by peer (error 104)
2026-06-19T06:38:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1896 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:38:50 dist-test-slave-nf4n krb5kdc[17341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:38:50 dist-test-slave-nf4n krb5kdc[17341](info): set up 2 sockets
Jun 19 06:38:50 dist-test-slave-nf4n krb5kdc[17341](info): commencing operation
krb5kdc: starting...
W20260619 06:38:51.345161 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:38:51 dist-test-slave-nf4n krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:38:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:38:51.588131 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:51.911552 17354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:51.911887 17354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:51.912041 17354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:51.919165 17354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:51.919332 17354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:51.919422 17354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:51.919539 17354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:51.919629 17354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:51.929224 17354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43319
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:51.930500 17354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:51.931807 17354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:51.943545 17360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:51.943470 17359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:51.943519 17362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:51.944324 17354 server_base.cc:1034] running on GCE node
I20260619 06:38:51.945151 17354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:51.946574 17354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:51.947780 17354 hybrid_clock.cc:648] HybridClock initialized: now 1781851131947705 us; error 64 us; skew 500 ppm
I20260619 06:38:51.948136 17354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:51 dist-test-slave-nf4n krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:51.954555 17354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:38:51.955974 17354 webserver.cc:460] Webserver started at http://127.0.0.1:37767/ using document root <none> and password file <none>
I20260619 06:38:51.956756 17354 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:51.956889 17354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:51.957257 17354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:51.960219 17354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "587d186117ac48168c8b3327d3e90889"
format_stamp: "Formatted at 2026-06-19 06:38:51 on dist-test-slave-nf4n"
server_key: "597f1db04ab43e4338ee64f8b9217f8c"
server_key_iv: "4adc60ceee83abdf55ff0f2022f377b9"
server_key_version: "encryptionkey@0"
I20260619 06:38:51.961087 17354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "587d186117ac48168c8b3327d3e90889"
format_stamp: "Formatted at 2026-06-19 06:38:51 on dist-test-slave-nf4n"
server_key: "597f1db04ab43e4338ee64f8b9217f8c"
server_key_iv: "4adc60ceee83abdf55ff0f2022f377b9"
server_key_version: "encryptionkey@0"
I20260619 06:38:51.966290 17354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:38:51.969702 17369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:51.971082 17354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:38:51.971283 17354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "587d186117ac48168c8b3327d3e90889"
format_stamp: "Formatted at 2026-06-19 06:38:51 on dist-test-slave-nf4n"
server_key: "597f1db04ab43e4338ee64f8b9217f8c"
server_key_iv: "4adc60ceee83abdf55ff0f2022f377b9"
server_key_version: "encryptionkey@0"
I20260619 06:38:51.971479 17354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:51.977111 17354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:51.981459 17354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:51.981742 17354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:52.002732 17354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43319
I20260619 06:38:52.002775 17420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43319 every 8 connection(s)
I20260619 06:38:52.004169 17354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:52.007819 17421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:52.011341 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 17354
I20260619 06:38:52.011518 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:52.011967 29091 external_mini_cluster.cc:1428] Setting key 7355379a609e146912c44ed2930b55a6
I20260619 06:38:52.020423 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:52.013844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46417 (local address 127.0.0.1:43319)
0619 06:38:52.014628 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:52.014643 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:52.014688 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:38:52.016575 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:52.016609 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:52.017071 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:52.017533 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:52.018544 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:52.018554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:52.018575 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:52.018592 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:52.018655 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:52.018685 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:52.018907 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:52.019042 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 06:38:52.019228 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":321,"threads_started":1}
I20260619 06:38:52.021277 17421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889: Bootstrap starting.
I20260619 06:38:52.023993 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:38:52.024554 17421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:52.025563 17421 log.cc:824] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:52.028751 17421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889: No bootstrap required, opened a new log
I20260619 06:38:52.031011 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:52.028497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51169 (local address 127.0.0.1:43319)
0619 06:38:52.028664 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:52.028670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:52.029064 (+ 394us) server_negotiation.cc:407] Connection header received
0619 06:38:52.029255 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:52.029262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:52.029323 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:52.029440 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:52.030342 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:52.030351 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:52.030356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:52.030375 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:52.030433 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:52.030457 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:52.030572 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:52.030648 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 06:38:52.030718 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:38:52.037400 17421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587d186117ac48168c8b3327d3e90889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } }
I20260619 06:38:52.037750 17421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:52.037861 17421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587d186117ac48168c8b3327d3e90889, State: Initialized, Role: FOLLOWER
I20260619 06:38:52.038408 17421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587d186117ac48168c8b3327d3e90889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } }
I20260619 06:38:52.038641 17421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:52.038794 17421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:52.038975 17421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:52.041539 17421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587d186117ac48168c8b3327d3e90889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } }
I20260619 06:38:52.041980 17421 leader_election.cc:302] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 587d186117ac48168c8b3327d3e90889; no voters:
I20260619 06:38:52.042474 17421 leader_election.cc:288] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:52.042620 17431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:52.043282 17431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [term 1 LEADER]: Becoming Leader. State: Replica: 587d186117ac48168c8b3327d3e90889, State: Running, Role: LEADER
I20260619 06:38:52.043745 17431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587d186117ac48168c8b3327d3e90889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } }
I20260619 06:38:52.044543 17421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:52.046715 17432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "587d186117ac48168c8b3327d3e90889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587d186117ac48168c8b3327d3e90889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } } }
I20260619 06:38:52.047091 17432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:52.046906 17433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 587d186117ac48168c8b3327d3e90889. Latest consensus state: current_term: 1 leader_uuid: "587d186117ac48168c8b3327d3e90889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587d186117ac48168c8b3327d3e90889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } } }
I20260619 06:38:52.047289 17433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:52.050251 17438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:52.055532 17438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:52.064432 17438 catalog_manager.cc:1347] Generated new cluster ID: 3f65f62b6603415180a04c16a4bd213a
I20260619 06:38:52.064534 17438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:52.088909 17438 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:52.089367 17438 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:52.094725 17438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 587d186117ac48168c8b3327d3e90889: Generated new TSK 0
I20260619 06:38:52.095633 17438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:38:52.111685 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17354
W20260619 06:38:52.112741 17416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46417
W20260619 06:38:52.113468 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:38:52.113621 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 17354
2026-06-19T06:38:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:38:52 dist-test-slave-nf4n krb5kdc[17452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:38:52 dist-test-slave-nf4n krb5kdc[17452](info): set up 2 sockets
Jun 19 06:38:52 dist-test-slave-nf4n krb5kdc[17452](info): commencing operation
krb5kdc: starting...
W20260619 06:38:53.237124 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:38:53 dist-test-slave-nf4n krb5kdc[17452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:38:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:38:53.485652 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:53.814500 17466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:53.814774 17466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:53.814944 17466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:53.822053 17466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:53.822153 17466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:53.822268 17466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:53.822369 17466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:53.822431 17466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:53.831909 17466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:53.833241 17466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:53.834545 17466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:53.846117 17471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:53.846174 17474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:53.846202 17472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:53.847422 17466 server_base.cc:1034] running on GCE node
I20260619 06:38:53.848311 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:53.849927 17466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:53.851155 17466 hybrid_clock.cc:648] HybridClock initialized: now 1781851133851111 us; error 34 us; skew 500 ppm
I20260619 06:38:53.851532 17466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:53 dist-test-slave-nf4n krb5kdc[17452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:53.854885 17466 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:38:53.856288 17466 webserver.cc:460] Webserver started at http://127.0.0.1:40521/ using document root <none> and password file <none>
I20260619 06:38:53.856899 17466 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:53.856997 17466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:53.857314 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:53.860520 17466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "86d691c90cb44de0beae76048489507f"
format_stamp: "Formatted at 2026-06-19 06:38:53 on dist-test-slave-nf4n"
server_key: "5b0606c9d9b5e5f786273090653362f8"
server_key_iv: "17912833b9270f8bb1f45eaa5dd77b2f"
server_key_version: "encryptionkey@0"
I20260619 06:38:53.861431 17466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "86d691c90cb44de0beae76048489507f"
format_stamp: "Formatted at 2026-06-19 06:38:53 on dist-test-slave-nf4n"
server_key: "5b0606c9d9b5e5f786273090653362f8"
server_key_iv: "17912833b9270f8bb1f45eaa5dd77b2f"
server_key_version: "encryptionkey@0"
I20260619 06:38:53.867087 17466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:38:53.870731 17481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:53.872072 17466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:38:53.872275 17466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "86d691c90cb44de0beae76048489507f"
format_stamp: "Formatted at 2026-06-19 06:38:53 on dist-test-slave-nf4n"
server_key: "5b0606c9d9b5e5f786273090653362f8"
server_key_iv: "17912833b9270f8bb1f45eaa5dd77b2f"
server_key_version: "encryptionkey@0"
I20260619 06:38:53.872498 17466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:53.909552 17466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:53.914427 17466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:53.914772 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:53.934702 17466 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33511
I20260619 06:38:53.934726 17532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33511 every 8 connection(s)
I20260619 06:38:53.936252 17466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:53.939461 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 17466
I20260619 06:38:53.939639 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:53.940191 29091 external_mini_cluster.cc:1428] Setting key 712c2ce3f39fcfddac0d1aba4f1948d2
I20260619 06:38:53.940488 17533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:53.952783 17533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f: Bootstrap starting.
Jun 19 06:38:53 dist-test-slave-nf4n krb5kdc[17452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:38:53.955806 17533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:53.956878 17533 log.cc:824] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:53.959689 17533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f: No bootstrap required, opened a new log
I20260619 06:38:53.962397 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:53.942086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58209 (local address 127.0.0.1:33511)
0619 06:38:53.942858 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:53.942870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:53.942910 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:38:53.945175 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:53.945196 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:53.945634 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:53.946027 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:53.947441 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:53.948684 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:53.950025 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:53.950717 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:53.950954 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:53.955067 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:53.955074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:53.955088 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:53.955122 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:53.958276 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:53.958984 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:53.958990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:53.958996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:53.959153 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:53.959567 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:53.959571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:53.959573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:53.960130 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:53.960457 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:53.960913 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 06:38:53.961410 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":347,"threads_started":1}
I20260619 06:38:53.966320 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:38:53.968221 17533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d691c90cb44de0beae76048489507f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } }
I20260619 06:38:53.968590 17533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:53.968719 17533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d691c90cb44de0beae76048489507f, State: Initialized, Role: FOLLOWER
I20260619 06:38:53.969480 17533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d691c90cb44de0beae76048489507f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } }
I20260619 06:38:53.969730 17533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:53.969868 17533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:53.970046 17533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:53.972821 17533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d691c90cb44de0beae76048489507f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } }
I20260619 06:38:53.973479 17533 leader_election.cc:302] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86d691c90cb44de0beae76048489507f; no voters:
I20260619 06:38:53.974110 17533 leader_election.cc:288] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:53.974227 17543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:53.976279 17533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:53.976660 17543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [term 1 LEADER]: Becoming Leader. State: Replica: 86d691c90cb44de0beae76048489507f, State: Running, Role: LEADER
I20260619 06:38:53.977311 17543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d691c90cb44de0beae76048489507f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } }
I20260619 06:38:53.980362 17544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86d691c90cb44de0beae76048489507f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d691c90cb44de0beae76048489507f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } } }
I20260619 06:38:53.980901 17544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:53.981163 17545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86d691c90cb44de0beae76048489507f. Latest consensus state: current_term: 1 leader_uuid: "86d691c90cb44de0beae76048489507f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86d691c90cb44de0beae76048489507f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33511 } } }
I20260619 06:38:53.982204 17548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:53.982343 17545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:53.989699 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:53.970663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59847 (local address 127.0.0.1:33511)
0619 06:38:53.970843 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:53.970847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:53.971324 (+ 477us) server_negotiation.cc:407] Connection header received
0619 06:38:53.971440 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:53.971445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:53.971525 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:53.971631 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:53.972925 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:53.974029 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:53.976384 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:53.977214 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:53.977337 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:53.980404 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:53.980409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:53.980412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:53.980440 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:53.984988 (+ 4548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:53.986049 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:53.986057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:53.986063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:53.986236 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:53.986760 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:53.986768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:53.986774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:53.987309 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:53.987642 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:53.988929 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0619 06:38:53.989335 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:38:53.993475 17548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:54.003566 17548 catalog_manager.cc:1347] Generated new cluster ID: 81dcbca33b544249b7448578ae775fe8
I20260619 06:38:54.003671 17548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:54.023871 17548 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:54.025256 17548 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:54.036175 17548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86d691c90cb44de0beae76048489507f: Generated new TSK 0
I20260619 06:38:54.037143 17548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:38:54.048055 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17466
I20260619 06:38:54.049371 17528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:54.140446 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 17466
2026-06-19T06:38:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:38:54 dist-test-slave-nf4n krb5kdc[17565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:38:54 dist-test-slave-nf4n krb5kdc[17565](info): set up 2 sockets
Jun 19 06:38:54 dist-test-slave-nf4n krb5kdc[17565](info): commencing operation
krb5kdc: starting...
W20260619 06:38:55.253176 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:38:55 dist-test-slave-nf4n krb5kdc[17565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:38:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:38:55.500660 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:55.823469 17578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:55.823756 17578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:55.823860 17578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:55.831063 17578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:55.831178 17578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:55.831225 17578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:55.831286 17578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:55.831333 17578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:55.841097 17578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:55.842355 17578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:55.843643 17578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:55.855747 17584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:55.855829 17583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:55.855824 17586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:55.856617 17578 server_base.cc:1034] running on GCE node
I20260619 06:38:55.857455 17578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:55.858829 17578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:55.860030 17578 hybrid_clock.cc:648] HybridClock initialized: now 1781851135859990 us; error 25 us; skew 500 ppm
I20260619 06:38:55.860340 17578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:55 dist-test-slave-nf4n krb5kdc[17565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:55.863436 17578 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:38:55.865000 17578 webserver.cc:460] Webserver started at http://127.0.0.1:41887/ using document root <none> and password file <none>
I20260619 06:38:55.865815 17578 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:55.865940 17578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:55.866245 17578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:55.869246 17578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "8ce286bcfa3944e2a5ce17a80a92630f"
format_stamp: "Formatted at 2026-06-19 06:38:55 on dist-test-slave-nf4n"
server_key: "e935cf992d77d6801b697e72f5d7b8a8"
server_key_iv: "c84ef7c6607dcd6f0feb49465690d81a"
server_key_version: "encryptionkey@0"
I20260619 06:38:55.869990 17578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "8ce286bcfa3944e2a5ce17a80a92630f"
format_stamp: "Formatted at 2026-06-19 06:38:55 on dist-test-slave-nf4n"
server_key: "e935cf992d77d6801b697e72f5d7b8a8"
server_key_iv: "c84ef7c6607dcd6f0feb49465690d81a"
server_key_version: "encryptionkey@0"
I20260619 06:38:55.875056 17578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:38:55.878396 17593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:55.879643 17578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:38:55.879805 17578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "8ce286bcfa3944e2a5ce17a80a92630f"
format_stamp: "Formatted at 2026-06-19 06:38:55 on dist-test-slave-nf4n"
server_key: "e935cf992d77d6801b697e72f5d7b8a8"
server_key_iv: "c84ef7c6607dcd6f0feb49465690d81a"
server_key_version: "encryptionkey@0"
I20260619 06:38:55.880093 17578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:55.894284 17578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:55.898041 17578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:55.898309 17578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:55.916746 17578 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43517
I20260619 06:38:55.916800 17645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43517 every 8 connection(s)
I20260619 06:38:55.918196 17578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:55.922109 17646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:55.923602 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 17578
I20260619 06:38:55.923790 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:55.924305 29091 external_mini_cluster.cc:1428] Setting key c31fe5b3075dfcaa31435458dffd9282
I20260619 06:38:55.935840 17646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f: Bootstrap starting.
I20260619 06:38:55.937232 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:55.926048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40513 (local address 127.0.0.1:43517)
0619 06:38:55.926896 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:55.926906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:55.926947 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:38:55.928469 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:55.928493 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:55.928875 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:55.929252 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:55.929818 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:55.931023 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:55.932937 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:55.933668 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:55.933809 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:55.935016 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:55.935023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:55.935035 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:55.935535 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:55.935604 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:55.935626 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:55.935707 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:55.935967 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 06:38:55.936257 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":442,"threads_started":1}
I20260619 06:38:55.938958 17646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:55.940100 17646 log.cc:824] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:55.940819 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:38:55.943027 17646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f: No bootstrap required, opened a new log
I20260619 06:38:55.951681 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:55.945068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52169 (local address 127.0.0.1:43517)
0619 06:38:55.945249 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:55.945256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:55.945688 (+ 432us) server_negotiation.cc:407] Connection header received
0619 06:38:55.945835 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:55.945840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:55.945928 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:55.946047 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:55.946500 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:55.947642 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:55.948849 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:55.949668 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:55.949804 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:55.950821 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:55.950828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:55.950834 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:55.950857 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:55.950915 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:55.950937 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:55.951070 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:55.951172 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 06:38:55.951277 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:38:55.953601 17646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce286bcfa3944e2a5ce17a80a92630f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43517 } }
I20260619 06:38:55.954005 17646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:55.954088 17646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce286bcfa3944e2a5ce17a80a92630f, State: Initialized, Role: FOLLOWER
I20260619 06:38:55.954658 17646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce286bcfa3944e2a5ce17a80a92630f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43517 } }
I20260619 06:38:55.954875 17646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:55.954988 17646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:55.955125 17646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:55.957727 17646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce286bcfa3944e2a5ce17a80a92630f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43517 } }
I20260619 06:38:55.958209 17646 leader_election.cc:302] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ce286bcfa3944e2a5ce17a80a92630f; no voters:
I20260619 06:38:55.958729 17646 leader_election.cc:288] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:55.958900 17656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:55.960070 17656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [term 1 LEADER]: Becoming Leader. State: Replica: 8ce286bcfa3944e2a5ce17a80a92630f, State: Running, Role: LEADER
I20260619 06:38:55.960637 17656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce286bcfa3944e2a5ce17a80a92630f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43517 } }
I20260619 06:38:55.961470 17646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:55.963409 17658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ce286bcfa3944e2a5ce17a80a92630f. Latest consensus state: current_term: 1 leader_uuid: "8ce286bcfa3944e2a5ce17a80a92630f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce286bcfa3944e2a5ce17a80a92630f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43517 } } }
I20260619 06:38:55.963783 17658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:55.964730 17662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:55.964529 17657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ce286bcfa3944e2a5ce17a80a92630f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce286bcfa3944e2a5ce17a80a92630f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43517 } } }
I20260619 06:38:55.966513 17657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:55.971612 17662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:55.980638 17662 catalog_manager.cc:1347] Generated new cluster ID: b41ef33799f1464a8696b97ef638d050
I20260619 06:38:55.980744 17662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:56.005999 17662 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:56.007452 17662 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:56.018625 17662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ce286bcfa3944e2a5ce17a80a92630f: Generated new TSK 0
I20260619 06:38:56.019466 17662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:38:56.029548 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17578
W20260619 06:38:56.030526 17641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40513
W20260619 06:38:56.031168 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17578: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:38:56.031316 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 17578
2026-06-19T06:38:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:38:56 dist-test-slave-nf4n krb5kdc[17677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:38:56 dist-test-slave-nf4n krb5kdc[17677](info): set up 2 sockets
Jun 19 06:38:56 dist-test-slave-nf4n krb5kdc[17677](info): commencing operation
krb5kdc: starting...
W20260619 06:38:57.136960 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:38:57 dist-test-slave-nf4n krb5kdc[17677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:38:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:38:57.400274 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:57.728814 17690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:57.729147 17690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:57.729303 17690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:57.736469 17690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:57.736627 17690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:57.736729 17690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:57.736835 17690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:57.736915 17690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:57.746794 17690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:57.748093 17690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:57.749528 17690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:57.760900 17695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:57.760900 17696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:57.761117 17698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:57.762140 17690 server_base.cc:1034] running on GCE node
I20260619 06:38:57.762964 17690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:57.764317 17690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:57.765532 17690 hybrid_clock.cc:648] HybridClock initialized: now 1781851137765500 us; error 49 us; skew 500 ppm
I20260619 06:38:57.765888 17690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:57 dist-test-slave-nf4n krb5kdc[17677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:57.769001 17690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:38:57.770457 17690 webserver.cc:460] Webserver started at http://127.0.0.1:45797/ using document root <none> and password file <none>
I20260619 06:38:57.771159 17690 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:57.771297 17690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:57.771641 17690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:57.774747 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "e77fa78816e14e2c9f36db0cd40f1333"
format_stamp: "Formatted at 2026-06-19 06:38:57 on dist-test-slave-nf4n"
server_key: "f91f1adf17020946e25b79dcf5dbfdf9"
server_key_iv: "ee99ab33b9d908b3e25822dd753a6693"
server_key_version: "encryptionkey@0"
I20260619 06:38:57.775513 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "e77fa78816e14e2c9f36db0cd40f1333"
format_stamp: "Formatted at 2026-06-19 06:38:57 on dist-test-slave-nf4n"
server_key: "f91f1adf17020946e25b79dcf5dbfdf9"
server_key_iv: "ee99ab33b9d908b3e25822dd753a6693"
server_key_version: "encryptionkey@0"
I20260619 06:38:57.780408 17690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:38:57.783870 17705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:57.785190 17690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:38:57.785379 17690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "e77fa78816e14e2c9f36db0cd40f1333"
format_stamp: "Formatted at 2026-06-19 06:38:57 on dist-test-slave-nf4n"
server_key: "f91f1adf17020946e25b79dcf5dbfdf9"
server_key_iv: "ee99ab33b9d908b3e25822dd753a6693"
server_key_version: "encryptionkey@0"
I20260619 06:38:57.785557 17690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:57.791013 17690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:57.795173 17690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:57.795451 17690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:57.812135 17690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37033
I20260619 06:38:57.812177 17757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37033 every 8 connection(s)
I20260619 06:38:57.813598 17690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:57.817366 17758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:57.822865 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 17690
I20260619 06:38:57.823038 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:57.823467 29091 external_mini_cluster.cc:1428] Setting key d33530f53d28236cc87153f6dff1d7d3
I20260619 06:38:57.827898 17758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333: Bootstrap starting.
I20260619 06:38:57.830981 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:57.825212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60111 (local address 127.0.0.1:37033)
0619 06:38:57.826020 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:57.826034 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:57.826081 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:38:57.827648 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:57.827670 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:57.827989 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:57.828362 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:57.829402 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:57.829409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:57.829420 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:57.829434 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:57.829493 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:57.829514 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:57.829683 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:57.829883 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 06:38:57.830082 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":378,"threads_started":1}
I20260619 06:38:57.831710 17758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:57.832886 17758 log.cc:824] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:57.834872 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:38:57.836071 17758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333: No bootstrap required, opened a new log
I20260619 06:38:57.841816 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:57.839238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60257 (local address 127.0.0.1:37033)
0619 06:38:57.839405 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:57.839420 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:57.839800 (+ 380us) server_negotiation.cc:407] Connection header received
0619 06:38:57.840001 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:57.840005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:57.840047 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:57.840162 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:57.841144 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:57.841150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:57.841160 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:38:57.841178 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:57.841231 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:38:57.841246 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:57.841386 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:57.841499 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 06:38:57.841561 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:38:57.844506 17758 raft_consensus.cc:348] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77fa78816e14e2c9f36db0cd40f1333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37033 } }
I20260619 06:38:57.844929 17758 raft_consensus.cc:374] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:57.845109 17758 raft_consensus.cc:729] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77fa78816e14e2c9f36db0cd40f1333, State: Initialized, Role: FOLLOWER
I20260619 06:38:57.845678 17758 consensus_queue.cc:260] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77fa78816e14e2c9f36db0cd40f1333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37033 } }
I20260619 06:38:57.845908 17758 raft_consensus.cc:388] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:57.846068 17758 raft_consensus.cc:482] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:57.846274 17758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:57.848410 17758 raft_consensus.cc:504] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77fa78816e14e2c9f36db0cd40f1333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37033 } }
I20260619 06:38:57.848940 17758 leader_election.cc:302] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e77fa78816e14e2c9f36db0cd40f1333; no voters:
I20260619 06:38:57.849535 17758 leader_election.cc:288] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:57.849663 17768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:57.850404 17768 raft_consensus.cc:686] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [term 1 LEADER]: Becoming Leader. State: Replica: e77fa78816e14e2c9f36db0cd40f1333, State: Running, Role: LEADER
I20260619 06:38:57.850970 17768 consensus_queue.cc:237] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77fa78816e14e2c9f36db0cd40f1333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37033 } }
I20260619 06:38:57.851629 17758 sys_catalog.cc:564] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:57.854024 17769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e77fa78816e14e2c9f36db0cd40f1333" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77fa78816e14e2c9f36db0cd40f1333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37033 } } }
I20260619 06:38:57.854063 17770 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e77fa78816e14e2c9f36db0cd40f1333. Latest consensus state: current_term: 1 leader_uuid: "e77fa78816e14e2c9f36db0cd40f1333" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77fa78816e14e2c9f36db0cd40f1333" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37033 } } }
I20260619 06:38:57.854476 17769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:57.854476 17770 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333 [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:57.856739 17775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:57.862365 17775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:57.871249 17775 catalog_manager.cc:1347] Generated new cluster ID: 9cdd0ef516c44a13bed9caebbaca7e2d
I20260619 06:38:57.871362 17775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:57.887229 17775 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:57.887620 17775 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:57.898134 17775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e77fa78816e14e2c9f36db0cd40f1333: Generated new TSK 0
I20260619 06:38:57.898857 17775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:38:57.906690 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17690
W20260619 06:38:57.907760 17752 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60111
W20260619 06:38:57.908378 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17690: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:38:57.908519 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 17690
2026-06-19T06:38:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1879 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:38:58 dist-test-slave-nf4n krb5kdc[17789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:38:58 dist-test-slave-nf4n krb5kdc[17789](info): set up 2 sockets
Jun 19 06:38:58 dist-test-slave-nf4n krb5kdc[17789](info): commencing operation
krb5kdc: starting...
W20260619 06:38:59.116973 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:38:59 dist-test-slave-nf4n krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:38:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:38:59.363353 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:38:59.690109 17802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:38:59.690445 17802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:38:59.690619 17802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:38:59.697747 17802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:38:59.697885 17802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:38:59.697983 17802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:38:59.698094 17802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:38:59.698181 17802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:38:59.707701 17802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:38:59.708959 17802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:38:59.710347 17802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:38:59.722250 17807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:59.722316 17808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:38:59.722335 17810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:38:59.723601 17802 server_base.cc:1034] running on GCE node
I20260619 06:38:59.724370 17802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:38:59.725854 17802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:38:59.727087 17802 hybrid_clock.cc:648] HybridClock initialized: now 1781851139727042 us; error 36 us; skew 500 ppm
I20260619 06:38:59.727444 17802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:38:59 dist-test-slave-nf4n krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:38:59.730545 17802 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:38:59.731977 17802 webserver.cc:460] Webserver started at http://127.0.0.1:42809/ using document root <none> and password file <none>
I20260619 06:38:59.732602 17802 fs_manager.cc:359] Metadata directory not provided
I20260619 06:38:59.732700 17802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:38:59.732939 17802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:38:59.735855 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "eba7162e502340829909eee916c5a7ff"
format_stamp: "Formatted at 2026-06-19 06:38:59 on dist-test-slave-nf4n"
server_key: "ad995d63b632524ca3c042e812ab3dad"
server_key_iv: "6730318798b156b1fdd8d6593243bfd4"
server_key_version: "encryptionkey@0"
I20260619 06:38:59.736605 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "eba7162e502340829909eee916c5a7ff"
format_stamp: "Formatted at 2026-06-19 06:38:59 on dist-test-slave-nf4n"
server_key: "ad995d63b632524ca3c042e812ab3dad"
server_key_iv: "6730318798b156b1fdd8d6593243bfd4"
server_key_version: "encryptionkey@0"
I20260619 06:38:59.741550 17802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:38:59.744799 17817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:38:59.746160 17802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:38:59.746294 17802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "eba7162e502340829909eee916c5a7ff"
format_stamp: "Formatted at 2026-06-19 06:38:59 on dist-test-slave-nf4n"
server_key: "ad995d63b632524ca3c042e812ab3dad"
server_key_iv: "6730318798b156b1fdd8d6593243bfd4"
server_key_version: "encryptionkey@0"
I20260619 06:38:59.746448 17802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:38:59.763278 17802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:38:59.767216 17802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:38:59.767442 17802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:38:59.790808 17802 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45167
I20260619 06:38:59.790856 17868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45167 every 8 connection(s)
I20260619 06:38:59.792620 17802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:38:59.797250 17869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:38:59.797356 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 17802
I20260619 06:38:59.797511 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:38:59.798044 29091 external_mini_cluster.cc:1428] Setting key 87b377499c18786689ea68c238811787
I20260619 06:38:59.811790 17869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff: Bootstrap starting.
Jun 19 06:38:59 dist-test-slave-nf4n krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:38:59.815721 17869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff: Neither blocks nor log segments found. Creating new log.
I20260619 06:38:59.816763 17869 log.cc:824] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff: Log is configured to *not* fsync() on all Append() calls
I20260619 06:38:59.819947 17869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff: No bootstrap required, opened a new log
I20260619 06:38:59.823711 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:59.800435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:44551 (local address 127.0.0.1:45167)
0619 06:38:59.801231 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:59.801248 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:59.801299 (+ 51us) server_negotiation.cc:407] Connection header received
0619 06:38:59.804047 (+ 2748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:59.804070 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:59.804480 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:59.804810 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:59.806601 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:59.807890 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:59.809729 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:59.810359 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:59.810482 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:59.815480 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:59.815488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:59.815502 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:59.815536 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:59.819223 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:59.820042 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:59.820049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:59.820057 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:59.820257 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:59.820685 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:59.820690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:59.820693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:59.821384 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:59.821724 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:59.822055 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 06:38:59.822395 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":434,"threads_started":1}
I20260619 06:38:59.827586 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:38:59.829355 17869 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba7162e502340829909eee916c5a7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260619 06:38:59.829691 17869 raft_consensus.cc:374] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:38:59.829784 17869 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba7162e502340829909eee916c5a7ff, State: Initialized, Role: FOLLOWER
I20260619 06:38:59.830469 17869 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba7162e502340829909eee916c5a7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260619 06:38:59.830737 17869 raft_consensus.cc:388] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:38:59.830868 17869 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:38:59.831032 17869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:38:59.833648 17869 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba7162e502340829909eee916c5a7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260619 06:38:59.834115 17869 leader_election.cc:302] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba7162e502340829909eee916c5a7ff; no voters:
I20260619 06:38:59.834839 17869 leader_election.cc:288] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:38:59.834923 17879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:38:59.836844 17869 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:38:59.837301 17879 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [term 1 LEADER]: Becoming Leader. State: Replica: eba7162e502340829909eee916c5a7ff, State: Running, Role: LEADER
I20260619 06:38:59.837929 17879 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba7162e502340829909eee916c5a7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260619 06:38:59.840950 17881 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba7162e502340829909eee916c5a7ff. Latest consensus state: current_term: 1 leader_uuid: "eba7162e502340829909eee916c5a7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba7162e502340829909eee916c5a7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } } }
I20260619 06:38:59.841408 17881 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:59.842062 17880 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eba7162e502340829909eee916c5a7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba7162e502340829909eee916c5a7ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } } }
I20260619 06:38:59.842458 17880 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff [sys.catalog]: This master's current role is: LEADER
I20260619 06:38:59.843487 17886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:38:59.848695 17886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:38:59.855070 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:38:59.831977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:59503 (local address 127.0.0.1:45167)
0619 06:38:59.832217 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:38:59.832226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:38:59.832781 (+ 555us) server_negotiation.cc:407] Connection header received
0619 06:38:59.832878 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:38:59.832884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:38:59.832955 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:38:59.833068 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:38:59.834503 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:59.835981 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:59.838072 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:38:59.838706 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:38:59.838822 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:38:59.844517 (+ 5695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:38:59.844522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:38:59.844525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:38:59.844560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:38:59.849380 (+ 4820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:59.850778 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:59.850786 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:59.850792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:59.850973 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:38:59.851709 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:38:59.851719 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:38:59.851726 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:38:59.852298 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:38:59.854063 (+ 1765us) server_negotiation.cc:1035] Waiting for connection context
0619 06:38:59.854290 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 06:38:59.854535 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:38:59.859023 17886 catalog_manager.cc:1347] Generated new cluster ID: 807f359df4a047cf9f1e220c7d713d51
I20260619 06:38:59.859148 17886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:38:59.873783 17886 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:38:59.875021 17886 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:38:59.885545 17886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eba7162e502340829909eee916c5a7ff: Generated new TSK 0
I20260619 06:38:59.886384 17886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:38:59.894102 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17802
I20260619 06:38:59.895454 17864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:38:59.986267 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 17802
W20260619 06:39:00.003906 17787 connection.cc:535] client connection to 127.0.0.1:45167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45167): Connection reset by peer (error 104)
2026-06-19T06:39:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2078 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:00 dist-test-slave-nf4n krb5kdc[17901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:00 dist-test-slave-nf4n krb5kdc[17901](info): set up 2 sockets
Jun 19 06:39:00 dist-test-slave-nf4n krb5kdc[17901](info): commencing operation
krb5kdc: starting...
W20260619 06:39:01.113095 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:01 dist-test-slave-nf4n krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:39:01.364228 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:01.687515 17914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:01.687855 17914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:01.688015 17914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:01.695147 17914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:01.695309 17914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:01.695413 17914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:01.695521 17914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:01.695604 17914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:01.705580 17914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:01.706890 17914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:01.708191 17914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:01.719745 17920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:01.719842 17919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:01.719817 17922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:01.720933 17914 server_base.cc:1034] running on GCE node
I20260619 06:39:01.721720 17914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:01.723054 17914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:01.724287 17914 hybrid_clock.cc:648] HybridClock initialized: now 1781851141724247 us; error 50 us; skew 500 ppm
I20260619 06:39:01.724632 17914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:01 dist-test-slave-nf4n krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:01.727762 17914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:39:01.729194 17914 webserver.cc:460] Webserver started at http://127.0.0.1:43339/ using document root <none> and password file <none>
I20260619 06:39:01.729803 17914 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:01.729900 17914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:01.730126 17914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:01.732944 17914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "566c51f53dfe4a12807582acc7c25288"
format_stamp: "Formatted at 2026-06-19 06:39:01 on dist-test-slave-nf4n"
server_key: "87b0e074e7dcf0a894923c280cb5e2d9"
server_key_iv: "75f8558f1bfb4254ece3f28fa61734fb"
server_key_version: "encryptionkey@0"
I20260619 06:39:01.733703 17914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "566c51f53dfe4a12807582acc7c25288"
format_stamp: "Formatted at 2026-06-19 06:39:01 on dist-test-slave-nf4n"
server_key: "87b0e074e7dcf0a894923c280cb5e2d9"
server_key_iv: "75f8558f1bfb4254ece3f28fa61734fb"
server_key_version: "encryptionkey@0"
I20260619 06:39:01.738536 17914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 06:39:01.741891 17929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:01.743186 17914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:39:01.743364 17914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "566c51f53dfe4a12807582acc7c25288"
format_stamp: "Formatted at 2026-06-19 06:39:01 on dist-test-slave-nf4n"
server_key: "87b0e074e7dcf0a894923c280cb5e2d9"
server_key_iv: "75f8558f1bfb4254ece3f28fa61734fb"
server_key_version: "encryptionkey@0"
I20260619 06:39:01.743527 17914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:01.769673 17914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:01.774175 17914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:01.774451 17914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:01.792630 17914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39201
I20260619 06:39:01.792662 17980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39201 every 8 connection(s)
I20260619 06:39:01.794063 17914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:01.797781 17981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:01.799438 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 17914
I20260619 06:39:01.799623 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:01.800079 29091 external_mini_cluster.cc:1428] Setting key ad9aca5ecdf6da82beb81602269fc8f3
I20260619 06:39:01.811868 17981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288: Bootstrap starting.
I20260619 06:39:01.814442 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:01.802055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:54915 (local address 127.0.0.1:39201)
0619 06:39:01.802859 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:01.802874 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:01.802921 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:39:01.804669 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:01.804697 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:01.805269 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:01.805669 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:01.806142 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:01.808053 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:01.809402 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:01.810394 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:01.810589 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:01.811714 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:01.811725 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:01.811749 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:01.812456 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:01.812548 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:01.812583 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:01.812745 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:01.812931 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 06:39:01.813135 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":310,"threads_started":1}
I20260619 06:39:01.815624 17981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:01.816689 17981 log.cc:824] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:01.818279 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:01.819552 17981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288: No bootstrap required, opened a new log
I20260619 06:39:01.828218 17981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566c51f53dfe4a12807582acc7c25288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260619 06:39:01.828629 17981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:01.828747 17981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 566c51f53dfe4a12807582acc7c25288, State: Initialized, Role: FOLLOWER
I20260619 06:39:01.829492 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:01.822836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:44823 (local address 127.0.0.1:39201)
0619 06:39:01.823017 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:01.823024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:01.823457 (+ 433us) server_negotiation.cc:407] Connection header received
0619 06:39:01.823603 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:01.823609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:01.823691 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:01.823803 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:01.824216 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:01.825398 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:01.826705 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:01.827401 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:01.827523 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:01.828636 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:01.828644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:01.828652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:01.828675 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:01.828727 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:01.828751 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:01.828873 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:01.829122 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 06:39:01.829175 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:39:01.829435 17981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566c51f53dfe4a12807582acc7c25288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260619 06:39:01.829746 17981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:01.829917 17981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:01.830096 17981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:01.832321 17981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566c51f53dfe4a12807582acc7c25288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260619 06:39:01.832844 17981 leader_election.cc:302] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 566c51f53dfe4a12807582acc7c25288; no voters:
I20260619 06:39:01.833448 17981 leader_election.cc:288] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:01.833575 17991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:01.834254 17991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [term 1 LEADER]: Becoming Leader. State: Replica: 566c51f53dfe4a12807582acc7c25288, State: Running, Role: LEADER
I20260619 06:39:01.834811 17991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566c51f53dfe4a12807582acc7c25288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260619 06:39:01.835583 17981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:01.837733 17992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "566c51f53dfe4a12807582acc7c25288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566c51f53dfe4a12807582acc7c25288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } } }
I20260619 06:39:01.838155 17992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:01.838038 17993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 566c51f53dfe4a12807582acc7c25288. Latest consensus state: current_term: 1 leader_uuid: "566c51f53dfe4a12807582acc7c25288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566c51f53dfe4a12807582acc7c25288" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } } }
I20260619 06:39:01.838374 17993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:01.842149 17999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:01.847277 17999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:01.855715 17999 catalog_manager.cc:1347] Generated new cluster ID: 10442fd03ef64e00b28410b4029a1ad7
I20260619 06:39:01.855898 17999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:01.877285 17999 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:01.879269 17999 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:01.888029 17999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 566c51f53dfe4a12807582acc7c25288: Generated new TSK 0
I20260619 06:39:01.889003 17999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:01.894953 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17914
W20260619 06:39:01.896811 17976 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:54915
W20260619 06:39:01.897614 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 17914: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:01.897760 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 17914
W20260619 06:39:01.912323 17899 connection.cc:535] client connection to 127.0.0.1:39201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39201): Connection reset by peer (error 104)
2026-06-19T06:39:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:01 dist-test-slave-nf4n krb5kdc[18012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:01 dist-test-slave-nf4n krb5kdc[18012](info): set up 2 sockets
Jun 19 06:39:01 dist-test-slave-nf4n krb5kdc[18012](info): commencing operation
krb5kdc: starting...
W20260619 06:39:03.009262 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:03 dist-test-slave-nf4n krb5kdc[18012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:39:03.256448 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:03.582082 18025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:03.582370 18025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:03.582564 18025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:03.589956 18025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:03.590058 18025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:03.590181 18025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:03.590288 18025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:03.590372 18025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:03.600183 18025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:03.601817 18025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:03.603255 18025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:03.615013 18030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:03.615090 18031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:03.615038 18034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:03.615788 18025 server_base.cc:1034] running on GCE node
I20260619 06:39:03.616547 18025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:03.618083 18025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:03.619314 18025 hybrid_clock.cc:648] HybridClock initialized: now 1781851143619254 us; error 53 us; skew 500 ppm
I20260619 06:39:03.619676 18025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:03 dist-test-slave-nf4n krb5kdc[18012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:03.627347 18025 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:39:03.628827 18025 webserver.cc:460] Webserver started at http://127.0.0.1:41519/ using document root <none> and password file <none>
I20260619 06:39:03.629534 18025 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:03.629632 18025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:03.629878 18025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:03.632941 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "b9379b8b036f477d885c9418bb89ee45"
format_stamp: "Formatted at 2026-06-19 06:39:03 on dist-test-slave-nf4n"
server_key: "0176e1e8d3b666e9af9013a943676be9"
server_key_iv: "40533efeae14a1f72bcfbd4fcdd950fe"
server_key_version: "encryptionkey@0"
I20260619 06:39:03.633736 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "b9379b8b036f477d885c9418bb89ee45"
format_stamp: "Formatted at 2026-06-19 06:39:03 on dist-test-slave-nf4n"
server_key: "0176e1e8d3b666e9af9013a943676be9"
server_key_iv: "40533efeae14a1f72bcfbd4fcdd950fe"
server_key_version: "encryptionkey@0"
I20260619 06:39:03.638608 18025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:39:03.641952 18041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:03.643239 18025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:03.643416 18025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "b9379b8b036f477d885c9418bb89ee45"
format_stamp: "Formatted at 2026-06-19 06:39:03 on dist-test-slave-nf4n"
server_key: "0176e1e8d3b666e9af9013a943676be9"
server_key_iv: "40533efeae14a1f72bcfbd4fcdd950fe"
server_key_version: "encryptionkey@0"
I20260619 06:39:03.643637 18025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:03.649204 18025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:03.652869 18025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:03.653249 18025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:03.671522 18025 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36829
I20260619 06:39:03.671566 18093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36829 every 8 connection(s)
I20260619 06:39:03.672914 18025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:03.676481 18094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:03.679040 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 18025
I20260619 06:39:03.679227 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:03.679641 29091 external_mini_cluster.cc:1428] Setting key 2b5ccbc2f99c4cc385ba3983694d41c3
I20260619 06:39:03.687045 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:03.681384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:46107 (local address 127.0.0.1:36829)
0619 06:39:03.682165 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:03.682176 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:03.682221 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:39:03.683823 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:03.683843 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:03.684152 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:03.684478 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:03.685559 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:03.685565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:03.685577 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:03.685590 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:03.685649 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:03.685671 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:03.685843 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:03.685955 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 06:39:03.686162 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":370,"threads_started":1}
I20260619 06:39:03.688181 18094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45: Bootstrap starting.
I20260619 06:39:03.690647 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:03.691478 18094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:03.692484 18094 log.cc:824] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:03.695711 18094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45: No bootstrap required, opened a new log
I20260619 06:39:03.697497 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:03.694949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:47429 (local address 127.0.0.1:36829)
0619 06:39:03.695089 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:03.695093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:03.695526 (+ 433us) server_negotiation.cc:407] Connection header received
0619 06:39:03.695717 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:03.695729 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:03.695772 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:03.695877 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:03.696793 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:03.696796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:03.696799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:03.696813 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:03.696846 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:03.696860 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:03.696944 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:03.697127 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 06:39:03.697171 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:39:03.703807 18094 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9379b8b036f477d885c9418bb89ee45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } }
I20260619 06:39:03.704149 18094 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:03.704255 18094 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9379b8b036f477d885c9418bb89ee45, State: Initialized, Role: FOLLOWER
I20260619 06:39:03.704782 18094 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9379b8b036f477d885c9418bb89ee45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } }
I20260619 06:39:03.705072 18094 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:03.705250 18094 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:03.705422 18094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:03.707960 18094 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9379b8b036f477d885c9418bb89ee45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } }
I20260619 06:39:03.708446 18094 leader_election.cc:302] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9379b8b036f477d885c9418bb89ee45; no voters:
I20260619 06:39:03.709097 18094 leader_election.cc:288] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:03.709309 18104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:03.710129 18104 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [term 1 LEADER]: Becoming Leader. State: Replica: b9379b8b036f477d885c9418bb89ee45, State: Running, Role: LEADER
I20260619 06:39:03.710706 18104 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9379b8b036f477d885c9418bb89ee45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } }
I20260619 06:39:03.711215 18094 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:03.713425 18106 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9379b8b036f477d885c9418bb89ee45. Latest consensus state: current_term: 1 leader_uuid: "b9379b8b036f477d885c9418bb89ee45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9379b8b036f477d885c9418bb89ee45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } } }
I20260619 06:39:03.713888 18106 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:03.714485 18105 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9379b8b036f477d885c9418bb89ee45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9379b8b036f477d885c9418bb89ee45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } } }
I20260619 06:39:03.714854 18105 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:03.716153 18110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:03.721879 18110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:03.730865 18110 catalog_manager.cc:1347] Generated new cluster ID: 9a6d2ad556d242da818960d8a5b6d45f
I20260619 06:39:03.730986 18110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:03.752662 18110 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:03.753131 18110 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:03.764236 18110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9379b8b036f477d885c9418bb89ee45: Generated new TSK 0
I20260619 06:39:03.765079 18110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:03.778908 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18025
W20260619 06:39:03.779983 18089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:46107
W20260619 06:39:03.780738 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18025: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:03.780891 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 18025
2026-06-19T06:39:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:03 dist-test-slave-nf4n krb5kdc[18125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:03 dist-test-slave-nf4n krb5kdc[18125](info): set up 2 sockets
Jun 19 06:39:03 dist-test-slave-nf4n krb5kdc[18125](info): commencing operation
krb5kdc: starting...
W20260619 06:39:04.893006 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:05 dist-test-slave-nf4n krb5kdc[18125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:39:05.137244 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:05.460338 18138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:05.460635 18138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:05.460738 18138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:05.468735 18138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:05.468852 18138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:05.468899 18138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:05.468958 18138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:05.469002 18138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:05.478958 18138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:05.480191 18138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:05.481520 18138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:05.493229 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:05.493247 18147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:05.493124 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:05.494328 18138 server_base.cc:1034] running on GCE node
I20260619 06:39:05.495098 18138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:05.496464 18138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:05.497699 18138 hybrid_clock.cc:648] HybridClock initialized: now 1781851145497647 us; error 53 us; skew 500 ppm
I20260619 06:39:05.498049 18138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:05 dist-test-slave-nf4n krb5kdc[18125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:05.500983 18138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:39:05.502481 18138 webserver.cc:460] Webserver started at http://127.0.0.1:43235/ using document root <none> and password file <none>
I20260619 06:39:05.503151 18138 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:05.503296 18138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:05.503589 18138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:05.506592 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "45f71cc197b441a5a635f8882326cf59"
format_stamp: "Formatted at 2026-06-19 06:39:05 on dist-test-slave-nf4n"
server_key: "71ea0010e7197c1fc47edd66e0112362"
server_key_iv: "034a2894a609bef5357c31670107bce3"
server_key_version: "encryptionkey@0"
I20260619 06:39:05.507354 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "45f71cc197b441a5a635f8882326cf59"
format_stamp: "Formatted at 2026-06-19 06:39:05 on dist-test-slave-nf4n"
server_key: "71ea0010e7197c1fc47edd66e0112362"
server_key_iv: "034a2894a609bef5357c31670107bce3"
server_key_version: "encryptionkey@0"
I20260619 06:39:05.512344 18138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 06:39:05.515646 18154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:05.516862 18138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:39:05.517091 18138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "45f71cc197b441a5a635f8882326cf59"
format_stamp: "Formatted at 2026-06-19 06:39:05 on dist-test-slave-nf4n"
server_key: "71ea0010e7197c1fc47edd66e0112362"
server_key_iv: "034a2894a609bef5357c31670107bce3"
server_key_version: "encryptionkey@0"
I20260619 06:39:05.517287 18138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:05.544891 18138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:05.549474 18138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:05.549819 18138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:05.569831 18138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33951
I20260619 06:39:05.569864 18205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33951 every 8 connection(s)
I20260619 06:39:05.571344 18138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:05.575188 18206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:05.579638 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 18138
I20260619 06:39:05.579831 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:05.580303 29091 external_mini_cluster.cc:1428] Setting key 5bc02a3acd335635ee54f74cca3b0948
I20260619 06:39:05.587920 18206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59: Bootstrap starting.
I20260619 06:39:05.591167 18206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:05.592232 18206 log.cc:824] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:39:05 dist-test-slave-nf4n krb5kdc[18125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:39:05.595052 18206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59: No bootstrap required, opened a new log
I20260619 06:39:05.602288 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:05.582218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:35973 (local address 127.0.0.1:33951)
0619 06:39:05.582964 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:05.582976 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:05.583016 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:39:05.584549 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:05.584575 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:05.585014 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:05.585477 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:05.586946 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:05.588320 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:05.589703 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:05.590462 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:05.590651 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:05.594848 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:05.594856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:05.594870 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:05.594904 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:05.598098 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:05.598899 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:05.598906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:05.598911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:05.599083 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:05.599571 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:05.599574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:05.599576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:05.600058 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:05.600400 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:05.600918 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 06:39:05.601291 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":306,"threads_started":1}
I20260619 06:39:05.603318 18206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f71cc197b441a5a635f8882326cf59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33951 } }
I20260619 06:39:05.603775 18206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:05.603904 18206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f71cc197b441a5a635f8882326cf59, State: Initialized, Role: FOLLOWER
I20260619 06:39:05.604467 18206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f71cc197b441a5a635f8882326cf59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33951 } }
I20260619 06:39:05.604717 18206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:05.604868 18206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:05.605090 18206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:05.606509 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:05.607530 18206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f71cc197b441a5a635f8882326cf59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33951 } }
I20260619 06:39:05.608031 18206 leader_election.cc:302] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f71cc197b441a5a635f8882326cf59; no voters:
I20260619 06:39:05.608697 18206 leader_election.cc:288] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:05.608785 18211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:05.611353 18206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:05.615957 18211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [term 1 LEADER]: Becoming Leader. State: Replica: 45f71cc197b441a5a635f8882326cf59, State: Running, Role: LEADER
I20260619 06:39:05.616829 18211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f71cc197b441a5a635f8882326cf59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33951 } }
I20260619 06:39:05.620905 18215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f71cc197b441a5a635f8882326cf59. Latest consensus state: current_term: 1 leader_uuid: "45f71cc197b441a5a635f8882326cf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f71cc197b441a5a635f8882326cf59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33951 } } }
I20260619 06:39:05.621388 18215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:05.622084 18214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f71cc197b441a5a635f8882326cf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f71cc197b441a5a635f8882326cf59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33951 } } }
I20260619 06:39:05.622460 18214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:05.623157 18224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:05.630084 18224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:05.637271 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:05.617753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:46555 (local address 127.0.0.1:33951)
0619 06:39:05.617979 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:05.617988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:05.618499 (+ 511us) server_negotiation.cc:407] Connection header received
0619 06:39:05.618699 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:05.618705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:05.618797 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:05.618907 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:05.621871 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:05.623152 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:05.624458 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:05.625266 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:05.625387 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:05.628727 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:05.628735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:05.628740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:05.628775 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:05.633040 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:05.634021 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:05.634031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:05.634035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:05.634218 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:05.634795 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:05.634800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:05.634804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:05.635211 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:05.635486 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:05.636721 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0619 06:39:05.636890 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:39:05.643988 18224 catalog_manager.cc:1347] Generated new cluster ID: 05857355e6214a988f9d84e95d5aa3af
I20260619 06:39:05.644192 18224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:05.672180 18224 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:05.673856 18224 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:05.685739 18224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f71cc197b441a5a635f8882326cf59: Generated new TSK 0
I20260619 06:39:05.686877 18224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:05.698014 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18138
I20260619 06:39:05.699021 18201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:05.791762 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 18138
W20260619 06:39:05.810002 18123 connection.cc:535] client connection to 127.0.0.1:33951 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33951): Connection reset by peer (error 104)
2026-06-19T06:39:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:05 dist-test-slave-nf4n krb5kdc[18239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:05 dist-test-slave-nf4n krb5kdc[18239](info): set up 2 sockets
Jun 19 06:39:05 dist-test-slave-nf4n krb5kdc[18239](info): commencing operation
krb5kdc: starting...
W20260619 06:39:06.893297 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:07 dist-test-slave-nf4n krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:39:07.144124 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:07.483943 18252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:07.484263 18252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:07.484440 18252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:07.491511 18252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:07.491654 18252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:07.491772 18252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:07.491875 18252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:07.491957 18252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:07.501806 18252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:07.503137 18252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:07.504434 18252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:07.515878 18258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:07.515954 18257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:07.516068 18260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:07.517081 18252 server_base.cc:1034] running on GCE node
I20260619 06:39:07.517858 18252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:07.519192 18252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:07.520411 18252 hybrid_clock.cc:648] HybridClock initialized: now 1781851147520375 us; error 51 us; skew 500 ppm
I20260619 06:39:07.520730 18252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:07 dist-test-slave-nf4n krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:07.523806 18252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:39:07.525345 18252 webserver.cc:460] Webserver started at http://127.0.0.1:39839/ using document root <none> and password file <none>
I20260619 06:39:07.525993 18252 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:07.526135 18252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:07.526423 18252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:07.529336 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "ab1f3a0e06034e47bf8afc13555df7df"
format_stamp: "Formatted at 2026-06-19 06:39:07 on dist-test-slave-nf4n"
server_key: "906ee110dd9d80b40a26bc09da944a80"
server_key_iv: "1f1889eb8523505ab4e848b9a09476fa"
server_key_version: "encryptionkey@0"
I20260619 06:39:07.530069 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "ab1f3a0e06034e47bf8afc13555df7df"
format_stamp: "Formatted at 2026-06-19 06:39:07 on dist-test-slave-nf4n"
server_key: "906ee110dd9d80b40a26bc09da944a80"
server_key_iv: "1f1889eb8523505ab4e848b9a09476fa"
server_key_version: "encryptionkey@0"
I20260619 06:39:07.535012 18252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 06:39:07.538304 18267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:07.539539 18252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:07.539719 18252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "ab1f3a0e06034e47bf8afc13555df7df"
format_stamp: "Formatted at 2026-06-19 06:39:07 on dist-test-slave-nf4n"
server_key: "906ee110dd9d80b40a26bc09da944a80"
server_key_iv: "1f1889eb8523505ab4e848b9a09476fa"
server_key_version: "encryptionkey@0"
I20260619 06:39:07.539929 18252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:07.551431 18252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:07.555707 18252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:07.556039 18252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:07.574494 18252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41827
I20260619 06:39:07.574523 18318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41827 every 8 connection(s)
I20260619 06:39:07.575937 18252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:07.579559 18319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:07.586206 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 18252
I20260619 06:39:07.586409 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:07.586824 29091 external_mini_cluster.cc:1428] Setting key ba44cb3af7b7aa9e200c9623f0be60aa
I20260619 06:39:07.589944 18319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df: Bootstrap starting.
I20260619 06:39:07.593297 18319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:07.594429 18319 log.cc:824] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:07.597391 18319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df: No bootstrap required, opened a new log
I20260619 06:39:07.599965 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:07.588677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:47345 (local address 127.0.0.1:41827)
0619 06:39:07.589522 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:07.589533 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:07.589572 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:39:07.591275 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:07.591301 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:07.591692 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:07.592006 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:07.592654 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:07.594039 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:07.595736 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:07.596363 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:07.596482 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:07.597757 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:07.597764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:07.597777 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:07.598308 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:07.598373 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:07.598396 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:07.598512 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:07.598785 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:39:07.598992 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":439,"threads_started":1}
I20260619 06:39:07.603694 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:07.605403 18319 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f3a0e06034e47bf8afc13555df7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } }
I20260619 06:39:07.605783 18319 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:07.605876 18319 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1f3a0e06034e47bf8afc13555df7df, State: Initialized, Role: FOLLOWER
I20260619 06:39:07.606508 18319 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f3a0e06034e47bf8afc13555df7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } }
I20260619 06:39:07.606745 18319 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:07.606894 18319 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:07.607089 18319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:07.610055 18319 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f3a0e06034e47bf8afc13555df7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } }
I20260619 06:39:07.610636 18319 leader_election.cc:302] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab1f3a0e06034e47bf8afc13555df7df; no voters:
I20260619 06:39:07.611307 18319 leader_election.cc:288] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:07.611389 18329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:07.612723 18329 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [term 1 LEADER]: Becoming Leader. State: Replica: ab1f3a0e06034e47bf8afc13555df7df, State: Running, Role: LEADER
I20260619 06:39:07.613363 18329 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f3a0e06034e47bf8afc13555df7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } }
I20260619 06:39:07.613644 18319 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:07.616146 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:07.608190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:60253 (local address 127.0.0.1:41827)
0619 06:39:07.608406 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:07.608414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:07.608816 (+ 402us) server_negotiation.cc:407] Connection header received
0619 06:39:07.608966 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:07.608969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:07.609092 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:07.609193 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:07.609714 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:07.610506 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:07.612470 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:07.613209 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:07.613352 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:07.614763 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:07.614772 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:07.614780 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:07.614813 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:07.614912 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:07.614940 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:07.615075 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:07.615610 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 06:39:07.615677 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:39:07.622927 18330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab1f3a0e06034e47bf8afc13555df7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f3a0e06034e47bf8afc13555df7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } } }
I20260619 06:39:07.622920 18331 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab1f3a0e06034e47bf8afc13555df7df. Latest consensus state: current_term: 1 leader_uuid: "ab1f3a0e06034e47bf8afc13555df7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f3a0e06034e47bf8afc13555df7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41827 } } }
I20260619 06:39:07.623382 18330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:07.623394 18331 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df [sys.catalog]: This master's current role is: LEADER
W20260619 06:39:07.628492 18343 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:39:07.628618 18343 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:39:07.628754 18344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:07.630354 18344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:07.638213 18344 catalog_manager.cc:1347] Generated new cluster ID: 0169625c8a63439b9add1ce5a9c4b318
I20260619 06:39:07.638315 18344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:07.650332 18344 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:07.651463 18344 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:07.666321 18344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab1f3a0e06034e47bf8afc13555df7df: Generated new TSK 0
I20260619 06:39:07.667147 18344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:07.673852 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18252
W20260619 06:39:07.675282 18314 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:47345
W20260619 06:39:07.676002 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18252: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:07.676149 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 18252
W20260619 06:39:07.693135 18237 connection.cc:535] client connection to 127.0.0.1:41827 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41827): Connection reset by peer (error 104)
2026-06-19T06:39:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:07 dist-test-slave-nf4n krb5kdc[18350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:07 dist-test-slave-nf4n krb5kdc[18350](info): set up 2 sockets
Jun 19 06:39:07 dist-test-slave-nf4n krb5kdc[18350](info): commencing operation
krb5kdc: starting...
W20260619 06:39:08.793138 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:08 dist-test-slave-nf4n krb5kdc[18350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:39:09.038209 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:09.360394 18363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:09.360682 18363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:09.360776 18363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:09.367955 18363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:09.368062 18363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:09.368108 18363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:09.368167 18363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:09.368203 18363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:09.378090 18363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:09.379307 18363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:09.380524 18363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:09.392210 18369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:09.392298 18371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:09.392230 18368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:09.393052 18363 server_base.cc:1034] running on GCE node
I20260619 06:39:09.393822 18363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:09.395196 18363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:09.396420 18363 hybrid_clock.cc:648] HybridClock initialized: now 1781851149396379 us; error 62 us; skew 500 ppm
I20260619 06:39:09.396842 18363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:09 dist-test-slave-nf4n krb5kdc[18350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:09.403836 18363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:39:09.405323 18363 webserver.cc:460] Webserver started at http://127.0.0.1:46413/ using document root <none> and password file <none>
I20260619 06:39:09.406018 18363 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:09.406169 18363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:09.406471 18363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:09.409461 18363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f"
format_stamp: "Formatted at 2026-06-19 06:39:09 on dist-test-slave-nf4n"
server_key: "09cee4a26cc6a545fe950ea2d448424a"
server_key_iv: "8f1c6e6a3db538a05c9110596713d7fc"
server_key_version: "encryptionkey@0"
I20260619 06:39:09.410279 18363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f"
format_stamp: "Formatted at 2026-06-19 06:39:09 on dist-test-slave-nf4n"
server_key: "09cee4a26cc6a545fe950ea2d448424a"
server_key_iv: "8f1c6e6a3db538a05c9110596713d7fc"
server_key_version: "encryptionkey@0"
I20260619 06:39:09.415577 18363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 06:39:09.419061 18379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:09.420470 18363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:39:09.420647 18363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f"
format_stamp: "Formatted at 2026-06-19 06:39:09 on dist-test-slave-nf4n"
server_key: "09cee4a26cc6a545fe950ea2d448424a"
server_key_iv: "8f1c6e6a3db538a05c9110596713d7fc"
server_key_version: "encryptionkey@0"
I20260619 06:39:09.420842 18363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:09.426370 18363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:09.430078 18363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:09.430363 18363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:09.447805 18363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35351
I20260619 06:39:09.447853 18430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35351 every 8 connection(s)
I20260619 06:39:09.449193 18363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:09.450591 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 18363
I20260619 06:39:09.450794 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:09.451340 29091 external_mini_cluster.cc:1428] Setting key 23e4ce8846ec8f6fd4bf2488fe626860
I20260619 06:39:09.453831 18431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:09.458946 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:09.453144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:34411 (local address 127.0.0.1:35351)
0619 06:39:09.453976 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:09.453994 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:09.454068 (+ 74us) server_negotiation.cc:407] Connection header received
0619 06:39:09.455653 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:09.455681 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:09.456004 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:09.456329 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:09.457389 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:09.457394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:09.457406 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:09.457420 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:09.457490 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:09.457516 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:09.457707 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:09.457842 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 06:39:09.458043 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":349,"threads_started":1}
I20260619 06:39:09.462414 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:09.466369 18431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f: Bootstrap starting.
I20260619 06:39:09.470233 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:09.467725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:56721 (local address 127.0.0.1:35351)
0619 06:39:09.467965 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:09.467973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:09.468306 (+ 333us) server_negotiation.cc:407] Connection header received
0619 06:39:09.468468 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:09.468471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:09.468516 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:09.468601 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:09.469595 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:09.469599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:09.469602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:09.469616 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:09.469652 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:09.469668 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:09.469750 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:09.469888 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 06:39:09.469934 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 06:39:09.470459 18431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:09.471509 18431 log.cc:824] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:09.474303 18431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f: No bootstrap required, opened a new log
I20260619 06:39:09.481778 18431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } }
I20260619 06:39:09.482172 18431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:09.482290 18431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ac11e0f62c40fa8b1f98c466b6dd5f, State: Initialized, Role: FOLLOWER
I20260619 06:39:09.482860 18431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } }
I20260619 06:39:09.483120 18431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:09.483280 18431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:09.483469 18431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:09.486290 18431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } }
I20260619 06:39:09.486899 18431 leader_election.cc:302] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37ac11e0f62c40fa8b1f98c466b6dd5f; no voters:
I20260619 06:39:09.487463 18431 leader_election.cc:288] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:09.487596 18441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:09.488308 18441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [term 1 LEADER]: Becoming Leader. State: Replica: 37ac11e0f62c40fa8b1f98c466b6dd5f, State: Running, Role: LEADER
I20260619 06:39:09.488885 18441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } }
I20260619 06:39:09.489624 18431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:09.491744 18442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } } }
I20260619 06:39:09.492138 18442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:09.498612 18443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37ac11e0f62c40fa8b1f98c466b6dd5f. Latest consensus state: current_term: 1 leader_uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ac11e0f62c40fa8b1f98c466b6dd5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } } }
I20260619 06:39:09.499305 18443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:09.500237 18449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:09.506148 18449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:09.515376 18449 catalog_manager.cc:1347] Generated new cluster ID: 86c7ae3893a34b37b545bbecbb635350
I20260619 06:39:09.515527 18449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:09.529637 18449 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:09.530083 18449 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:09.540030 18449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37ac11e0f62c40fa8b1f98c466b6dd5f: Generated new TSK 0
I20260619 06:39:09.540808 18449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:09.557499 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18363
W20260619 06:39:09.558645 18426 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.113:34411
W20260619 06:39:09.559391 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18363: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:09.559540 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 18363
2026-06-19T06:39:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1880 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:09 dist-test-slave-nf4n krb5kdc[18463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:09 dist-test-slave-nf4n krb5kdc[18463](info): set up 2 sockets
Jun 19 06:39:09 dist-test-slave-nf4n krb5kdc[18463](info): commencing operation
krb5kdc: starting...
W20260619 06:39:10.653282 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.011s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:10 dist-test-slave-nf4n krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:39:10.906100 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:32915
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:11.232376 18476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:11.232728 18476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:11.232883 18476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:11.240228 18476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:11.240381 18476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:11.240489 18476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:11.240594 18476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:11.240680 18476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:11.250586 18476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:32915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:11.251892 18476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:11.253270 18476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:11.264880 18484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:11.264871 18482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:11.265419 18481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:11.265676 18476 server_base.cc:1034] running on GCE node
I20260619 06:39:11.266377 18476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:11.267632 18476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:11.268869 18476 hybrid_clock.cc:648] HybridClock initialized: now 1781851151268829 us; error 51 us; skew 500 ppm
I20260619 06:39:11.269284 18476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:11 dist-test-slave-nf4n krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:11.272222 18476 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:39:11.273667 18476 webserver.cc:460] Webserver started at http://127.28.104.254:38505/ using document root <none> and password file <none>
I20260619 06:39:11.274330 18476 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:11.274474 18476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:11.274770 18476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:11.277693 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "74f72de0082c431683fe04381419c80b"
format_stamp: "Formatted at 2026-06-19 06:39:11 on dist-test-slave-nf4n"
server_key: "89054c5497e58ef971c47948f3797146"
server_key_iv: "e4e8189bc4543d73b8f0532b4a2a1634"
server_key_version: "encryptionkey@0"
I20260619 06:39:11.278431 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "74f72de0082c431683fe04381419c80b"
format_stamp: "Formatted at 2026-06-19 06:39:11 on dist-test-slave-nf4n"
server_key: "89054c5497e58ef971c47948f3797146"
server_key_iv: "e4e8189bc4543d73b8f0532b4a2a1634"
server_key_version: "encryptionkey@0"
I20260619 06:39:11.283277 18476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:39:11.286527 18491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:11.287690 18476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:39:11.287865 18476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "74f72de0082c431683fe04381419c80b"
format_stamp: "Formatted at 2026-06-19 06:39:11 on dist-test-slave-nf4n"
server_key: "89054c5497e58ef971c47948f3797146"
server_key_iv: "e4e8189bc4543d73b8f0532b4a2a1634"
server_key_version: "encryptionkey@0"
I20260619 06:39:11.288050 18476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:11.345203 18476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:11.349200 18476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:11.349493 18476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:11.367975 18476 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:32915
I20260619 06:39:11.367990 18543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:32915 every 8 connection(s)
I20260619 06:39:11.369447 18476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:11.373114 18544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:11.379408 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 18476
I20260619 06:39:11.379595 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:11.380007 29091 external_mini_cluster.cc:1428] Setting key a32f667ebdcfa4d35bee5362d9535b6c
I20260619 06:39:11.383836 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b: Bootstrap starting.
I20260619 06:39:11.387694 18544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:11.388929 18544 log.cc:824] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:11.391932 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b: No bootstrap required, opened a new log
Jun 19 06:39:11 dist-test-slave-nf4n krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:39:11.401683 18544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f72de0082c431683fe04381419c80b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32915 } }
I20260619 06:39:11.402144 18544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:11.402233 18544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f72de0082c431683fe04381419c80b, State: Initialized, Role: FOLLOWER
I20260619 06:39:11.402935 18544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f72de0082c431683fe04381419c80b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32915 } }
I20260619 06:39:11.403221 18544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:11.403368 18544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:11.403558 18544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:11.405797 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:11.382093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45823 (local address 127.28.104.254:32915)
0619 06:39:11.383057 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:11.383073 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:11.383445 (+ 372us) server_negotiation.cc:407] Connection header received
0619 06:39:11.385946 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:11.385979 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:11.386465 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:11.386896 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:11.388187 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:11.390063 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:11.391330 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:11.392423 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:11.392616 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:11.396896 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:11.396907 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:11.396931 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:11.396972 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:11.400941 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:11.401765 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:11.401773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:11.401781 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:11.401970 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:11.402526 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:11.402531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:11.402534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:11.403242 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:11.403568 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:11.404098 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 06:39:11.404438 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":431,"threads_started":1}
I20260619 06:39:11.406728 18544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f72de0082c431683fe04381419c80b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32915 } }
I20260619 06:39:11.407279 18544 leader_election.cc:302] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f72de0082c431683fe04381419c80b; no voters:
I20260619 06:39:11.408525 18544 leader_election.cc:288] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:11.408717 18549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:11.410454 18544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:11.410964 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:11.413280 18549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [term 1 LEADER]: Becoming Leader. State: Replica: 74f72de0082c431683fe04381419c80b, State: Running, Role: LEADER
I20260619 06:39:11.413846 18549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f72de0082c431683fe04381419c80b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32915 } }
I20260619 06:39:11.421411 18550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f72de0082c431683fe04381419c80b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f72de0082c431683fe04381419c80b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32915 } } }
I20260619 06:39:11.421875 18550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:11.422453 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f72de0082c431683fe04381419c80b. Latest consensus state: current_term: 1 leader_uuid: "74f72de0082c431683fe04381419c80b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f72de0082c431683fe04381419c80b" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 32915 } } }
I20260619 06:39:11.422765 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:11.425318 18564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:11.430699 18564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:11.440915 18564 catalog_manager.cc:1347] Generated new cluster ID: 0ab9fe1486a540499a02d82cca85c578
I20260619 06:39:11.441052 18564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:11.442561 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:11.428184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41717 (local address 127.28.104.254:32915)
0619 06:39:11.428423 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:11.428432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:11.428968 (+ 536us) server_negotiation.cc:407] Connection header received
0619 06:39:11.429228 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:11.429233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:11.429294 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:11.429412 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:11.430790 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:11.431615 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:11.432845 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:11.433344 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:11.433500 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:11.435843 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:11.435848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:11.435851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:11.435880 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:11.439473 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:11.440217 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:11.440224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:11.440227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:11.440360 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:11.440798 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:11.440803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:11.440808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:11.441239 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:11.441519 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:11.441909 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 06:39:11.442103 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:39:11.468040 18564 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:11.469425 18564 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:11.487566 18564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f72de0082c431683fe04381419c80b: Generated new TSK 0
I20260619 06:39:11.488533 18564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:11.493968 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18476
I20260619 06:39:11.495042 18539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:11.587850 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 18476
W20260619 06:39:11.605495 18461 connection.cc:535] client connection to 127.28.104.254:32915 recv error: Network error: failed to read from TLS socket (remote: 127.28.104.254:32915): Connection reset by peer (error 104)
2026-06-19T06:39:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2031 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:11 dist-test-slave-nf4n krb5kdc[18577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:11 dist-test-slave-nf4n krb5kdc[18577](info): set up 2 sockets
Jun 19 06:39:11 dist-test-slave-nf4n krb5kdc[18577](info): commencing operation
krb5kdc: starting...
W20260619 06:39:12.697172 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:12 dist-test-slave-nf4n krb5kdc[18577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:39:12.957604 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:44081
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:37093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:44081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:13.288183 18590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:13.288457 18590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:13.288630 18590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:13.295881 18590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:13.295985 18590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:13.296103 18590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:13.296214 18590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:13.296298 18590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:13.306079 18590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:37093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:44081
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:44081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:13.307337 18590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:13.308617 18590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:13.320590 18595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:13.320490 18598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:13.320498 18596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:13.321750 18590 server_base.cc:1034] running on GCE node
I20260619 06:39:13.322470 18590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:13.323896 18590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:13.325124 18590 hybrid_clock.cc:648] HybridClock initialized: now 1781851153325068 us; error 50 us; skew 500 ppm
I20260619 06:39:13.325433 18590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:13 dist-test-slave-nf4n krb5kdc[18577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:13.328498 18590 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:39:13.329979 18590 webserver.cc:460] Webserver started at http://127.28.104.254:39365/ using document root <none> and password file <none>
I20260619 06:39:13.330663 18590 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:13.330817 18590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:13.331156 18590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:13.334244 18590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "8d58b763042c45d68743c9605c8e3861"
format_stamp: "Formatted at 2026-06-19 06:39:13 on dist-test-slave-nf4n"
server_key: "58bf8bcf52c99cb95f35be1d45a3473b"
server_key_iv: "31f3a16dae0856f2f6a712bd48938aa3"
server_key_version: "encryptionkey@0"
I20260619 06:39:13.335005 18590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "8d58b763042c45d68743c9605c8e3861"
format_stamp: "Formatted at 2026-06-19 06:39:13 on dist-test-slave-nf4n"
server_key: "58bf8bcf52c99cb95f35be1d45a3473b"
server_key_iv: "31f3a16dae0856f2f6a712bd48938aa3"
server_key_version: "encryptionkey@0"
I20260619 06:39:13.340001 18590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:39:13.343573 18605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:13.345137 18590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:39:13.345325 18590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "8d58b763042c45d68743c9605c8e3861"
format_stamp: "Formatted at 2026-06-19 06:39:13 on dist-test-slave-nf4n"
server_key: "58bf8bcf52c99cb95f35be1d45a3473b"
server_key_iv: "31f3a16dae0856f2f6a712bd48938aa3"
server_key_version: "encryptionkey@0"
I20260619 06:39:13.345530 18590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:13.362879 18590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:13.366683 18590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:13.366957 18590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:13.385279 18590 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:44081
I20260619 06:39:13.385324 18656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:44081 every 8 connection(s)
I20260619 06:39:13.386637 18590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:13.390359 18657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:13.393237 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 18590
I20260619 06:39:13.393409 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:13.393930 29091 external_mini_cluster.cc:1428] Setting key 7295a1e578e3b693751f94376f896d11
I20260619 06:39:13.402311 18657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861: Bootstrap starting.
I20260619 06:39:13.405514 18657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:13.406682 18657 log.cc:824] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:13.407297 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:13.395876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41393 (local address 127.28.104.254:44081)
0619 06:39:13.396727 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:13.396738 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:13.396780 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:39:13.398467 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:13.398489 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:13.398893 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:13.399415 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:13.400010 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:13.401385 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:13.402754 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:13.403418 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:13.403636 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:13.404869 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:13.404876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:13.404890 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:13.405474 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:13.405547 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:13.405596 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:13.405779 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:13.406059 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 06:39:13.406213 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":411,"threads_started":1}
I20260619 06:39:13.409763 18657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861: No bootstrap required, opened a new log
I20260619 06:39:13.411085 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:13.419564 18657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d58b763042c45d68743c9605c8e3861" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44081 } }
I20260619 06:39:13.420058 18657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:13.420172 18657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d58b763042c45d68743c9605c8e3861, State: Initialized, Role: FOLLOWER
I20260619 06:39:13.420826 18657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d58b763042c45d68743c9605c8e3861" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44081 } }
I20260619 06:39:13.421126 18657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:13.421284 18657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:13.421482 18657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:13.423887 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:13.417059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32907 (local address 127.28.104.254:44081)
0619 06:39:13.417292 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:13.417316 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:13.417697 (+ 381us) server_negotiation.cc:407] Connection header received
0619 06:39:13.417881 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:13.417885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:13.417948 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:13.418030 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:13.418643 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:13.419378 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:13.421237 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:13.421813 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:13.421906 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:13.423132 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:13.423137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:13.423140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:13.423156 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:13.423195 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:13.423210 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:13.423290 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:13.423497 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 06:39:13.423539 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:39:13.424618 18657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d58b763042c45d68743c9605c8e3861" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44081 } }
I20260619 06:39:13.425266 18657 leader_election.cc:302] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d58b763042c45d68743c9605c8e3861; no voters:
I20260619 06:39:13.425820 18657 leader_election.cc:288] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:13.425951 18667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:13.426741 18667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [term 1 LEADER]: Becoming Leader. State: Replica: 8d58b763042c45d68743c9605c8e3861, State: Running, Role: LEADER
I20260619 06:39:13.427237 18667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d58b763042c45d68743c9605c8e3861" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44081 } }
I20260619 06:39:13.427975 18657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:13.430028 18669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d58b763042c45d68743c9605c8e3861. Latest consensus state: current_term: 1 leader_uuid: "8d58b763042c45d68743c9605c8e3861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d58b763042c45d68743c9605c8e3861" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44081 } } }
I20260619 06:39:13.430444 18669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:13.432065 18675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:13.432024 18668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d58b763042c45d68743c9605c8e3861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d58b763042c45d68743c9605c8e3861" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 44081 } } }
I20260619 06:39:13.432356 18668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:13.437240 18675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:13.446702 18675 catalog_manager.cc:1347] Generated new cluster ID: 6d762a6bba8a4fda9e0c2c8d37f23d82
I20260619 06:39:13.446815 18675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:13.466723 18675 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:13.468019 18675 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:13.479317 18675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d58b763042c45d68743c9605c8e3861: Generated new TSK 0
I20260619 06:39:13.480167 18675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:13.490226 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18590
W20260619 06:39:13.491394 18652 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41393
W20260619 06:39:13.492146 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18590: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:13.492293 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 18590
W20260619 06:39:13.509281 18575 connection.cc:535] client connection to 127.28.104.254:44081 recv error: Network error: failed to read from TLS socket (remote: 127.28.104.254:44081): Connection reset by peer (error 104)
2026-06-19T06:39:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:13 dist-test-slave-nf4n krb5kdc[18689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:13 dist-test-slave-nf4n krb5kdc[18689](info): set up 2 sockets
Jun 19 06:39:13 dist-test-slave-nf4n krb5kdc[18689](info): commencing operation
krb5kdc: starting...
W20260619 06:39:14.601142 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:14 dist-test-slave-nf4n krb5kdc[18689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:39:14.857286 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:43251
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:34893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:43251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:15.183461 18702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:15.183741 18702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:15.183929 18702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:15.190974 18702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:15.191095 18702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:15.191181 18702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:15.191285 18702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:15.191349 18702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:15.200990 18702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:34893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:43251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:43251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:15.202333 18702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:15.203661 18702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:15.215978 18710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:15.216008 18707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:15.216032 18708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:15.216862 18702 server_base.cc:1034] running on GCE node
I20260619 06:39:15.217716 18702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:15.219218 18702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:15.220538 18702 hybrid_clock.cc:648] HybridClock initialized: now 1781851155220462 us; error 65 us; skew 500 ppm
I20260619 06:39:15.220925 18702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:15 dist-test-slave-nf4n krb5kdc[18689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:15.228402 18702 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:39:15.230073 18702 webserver.cc:460] Webserver started at http://127.28.104.254:40959/ using document root <none> and password file <none>
I20260619 06:39:15.230803 18702 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:15.230952 18702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:15.231272 18702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:15.234416 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "c58f1a7ca823471e869994032d262bc7"
format_stamp: "Formatted at 2026-06-19 06:39:15 on dist-test-slave-nf4n"
server_key: "6517d0b133b06db3cca74a93fe44f172"
server_key_iv: "4b10449340fc69464f33bafa3c79448b"
server_key_version: "encryptionkey@0"
I20260619 06:39:15.235170 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "c58f1a7ca823471e869994032d262bc7"
format_stamp: "Formatted at 2026-06-19 06:39:15 on dist-test-slave-nf4n"
server_key: "6517d0b133b06db3cca74a93fe44f172"
server_key_iv: "4b10449340fc69464f33bafa3c79448b"
server_key_version: "encryptionkey@0"
I20260619 06:39:15.240280 18702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:39:15.243691 18717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:15.245107 18702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:39:15.245299 18702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "c58f1a7ca823471e869994032d262bc7"
format_stamp: "Formatted at 2026-06-19 06:39:15 on dist-test-slave-nf4n"
server_key: "6517d0b133b06db3cca74a93fe44f172"
server_key_iv: "4b10449340fc69464f33bafa3c79448b"
server_key_version: "encryptionkey@0"
I20260619 06:39:15.245509 18702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:15.251155 18702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:15.255033 18702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:15.255330 18702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:15.273572 18702 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:43251
I20260619 06:39:15.273602 18768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:43251 every 8 connection(s)
I20260619 06:39:15.274955 18702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:15.278589 18769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:15.280906 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 18702
I20260619 06:39:15.281100 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:15.281647 29091 external_mini_cluster.cc:1428] Setting key 4f3dfa9b199a4799e68d60b9d46edb58
I20260619 06:39:15.289671 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:15.283618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33303 (local address 127.28.104.254:43251)
0619 06:39:15.284441 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:15.284454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:15.284497 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:39:15.286225 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:15.286248 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:15.286582 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:15.286967 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:15.288005 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:15.288011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:15.288027 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:15.288041 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:15.288098 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:15.288128 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:15.288369 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:15.288529 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 06:39:15.288708 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":391,"threads_started":1}
I20260619 06:39:15.290745 18769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7: Bootstrap starting.
I20260619 06:39:15.293398 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:15.294064 18769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:15.295046 18769 log.cc:824] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:15.298319 18769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7: No bootstrap required, opened a new log
I20260619 06:39:15.300381 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:15.297834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49375 (local address 127.28.104.254:43251)
0619 06:39:15.297981 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:15.297985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:15.298456 (+ 471us) server_negotiation.cc:407] Connection header received
0619 06:39:15.298646 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:15.298657 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:15.298707 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:15.298812 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:15.299722 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:15.299727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:15.299732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:15.299756 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:15.299791 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:15.299806 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:15.299940 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:15.300047 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 06:39:15.300111 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:39:15.306514 18769 raft_consensus.cc:348] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58f1a7ca823471e869994032d262bc7" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43251 } }
I20260619 06:39:15.306970 18769 raft_consensus.cc:374] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:15.307075 18769 raft_consensus.cc:729] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58f1a7ca823471e869994032d262bc7, State: Initialized, Role: FOLLOWER
I20260619 06:39:15.307614 18769 consensus_queue.cc:260] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58f1a7ca823471e869994032d262bc7" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43251 } }
I20260619 06:39:15.307857 18769 raft_consensus.cc:388] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:15.307994 18769 raft_consensus.cc:482] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:15.308171 18769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:15.310424 18769 raft_consensus.cc:504] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58f1a7ca823471e869994032d262bc7" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43251 } }
I20260619 06:39:15.310894 18769 leader_election.cc:302] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c58f1a7ca823471e869994032d262bc7; no voters:
I20260619 06:39:15.311655 18769 leader_election.cc:288] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:15.311753 18779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:15.312450 18779 raft_consensus.cc:686] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [term 1 LEADER]: Becoming Leader. State: Replica: c58f1a7ca823471e869994032d262bc7, State: Running, Role: LEADER
I20260619 06:39:15.313052 18779 consensus_queue.cc:237] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58f1a7ca823471e869994032d262bc7" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43251 } }
I20260619 06:39:15.313920 18769 sys_catalog.cc:564] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:15.316022 18780 sys_catalog.cc:455] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c58f1a7ca823471e869994032d262bc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58f1a7ca823471e869994032d262bc7" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43251 } } }
I20260619 06:39:15.316057 18781 sys_catalog.cc:455] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c58f1a7ca823471e869994032d262bc7. Latest consensus state: current_term: 1 leader_uuid: "c58f1a7ca823471e869994032d262bc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58f1a7ca823471e869994032d262bc7" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 43251 } } }
I20260619 06:39:15.316498 18780 sys_catalog.cc:458] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:15.316526 18781 sys_catalog.cc:458] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:15.318974 18785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:15.325364 18785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:15.335111 18785 catalog_manager.cc:1347] Generated new cluster ID: 39e3b3b5520d4890babaf9201cda6a36
I20260619 06:39:15.335249 18785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:15.368259 18785 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:15.368673 18785 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:15.376020 18785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c58f1a7ca823471e869994032d262bc7: Generated new TSK 0
I20260619 06:39:15.376813 18785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:15.387585 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18702
W20260619 06:39:15.388821 18764 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33303
W20260619 06:39:15.389472 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18702: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:15.389619 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 18702
2026-06-19T06:39:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:15 dist-test-slave-nf4n krb5kdc[18800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:15 dist-test-slave-nf4n krb5kdc[18800](info): set up 2 sockets
Jun 19 06:39:15 dist-test-slave-nf4n krb5kdc[18800](info): commencing operation
krb5kdc: starting...
W20260619 06:39:16.536958 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:16 dist-test-slave-nf4n krb5kdc[18800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:39:16.789913 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:42435
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:42435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:17.117450 18813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:17.117832 18813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:17.118012 18813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:17.125418 18813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:17.125677 18813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:17.125854 18813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:17.125933 18813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:17.125980 18813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:17.135931 18813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:42435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:42435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:17.137174 18813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:17.138407 18813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:17.150259 18818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:17.150311 18821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:17.150362 18819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:17.151254 18813 server_base.cc:1034] running on GCE node
I20260619 06:39:17.151961 18813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:17.153438 18813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:17.154675 18813 hybrid_clock.cc:648] HybridClock initialized: now 1781851157154599 us; error 32 us; skew 500 ppm
I20260619 06:39:17.155030 18813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:17 dist-test-slave-nf4n krb5kdc[18800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:17.158130 18813 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:39:17.159610 18813 webserver.cc:460] Webserver started at http://127.28.104.254:44545/ using document root <none> and password file <none>
I20260619 06:39:17.160311 18813 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:17.160454 18813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:17.160765 18813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:17.163760 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "707690505f2b44a79f6b99a78df323f5"
format_stamp: "Formatted at 2026-06-19 06:39:17 on dist-test-slave-nf4n"
server_key: "1fe95b5e6beb31f24b7c9816a3b1ef96"
server_key_iv: "91221907c0be1e25c3cdeedf7fdf74bb"
server_key_version: "encryptionkey@0"
I20260619 06:39:17.164525 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "707690505f2b44a79f6b99a78df323f5"
format_stamp: "Formatted at 2026-06-19 06:39:17 on dist-test-slave-nf4n"
server_key: "1fe95b5e6beb31f24b7c9816a3b1ef96"
server_key_iv: "91221907c0be1e25c3cdeedf7fdf74bb"
server_key_version: "encryptionkey@0"
I20260619 06:39:17.169677 18813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:39:17.173096 18828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:17.174440 18813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:17.174630 18813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "707690505f2b44a79f6b99a78df323f5"
format_stamp: "Formatted at 2026-06-19 06:39:17 on dist-test-slave-nf4n"
server_key: "1fe95b5e6beb31f24b7c9816a3b1ef96"
server_key_iv: "91221907c0be1e25c3cdeedf7fdf74bb"
server_key_version: "encryptionkey@0"
I20260619 06:39:17.174846 18813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:17.194624 18813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:17.198482 18813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:17.198773 18813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:17.217490 18813 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:42435
I20260619 06:39:17.217514 18879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:42435 every 8 connection(s)
I20260619 06:39:17.218897 18813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:17.222731 18880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:17.223968 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 18813
I20260619 06:39:17.224144 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:17.224720 29091 external_mini_cluster.cc:1428] Setting key 35c3717441c11bd86156b23c899bc5bc
I20260619 06:39:17.236961 18880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5: Bootstrap starting.
Jun 19 06:39:17 dist-test-slave-nf4n krb5kdc[18800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:39:17.240833 18880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:17.241971 18880 log.cc:824] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:17.244741 18880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5: No bootstrap required, opened a new log
I20260619 06:39:17.248054 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:17.226575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34293 (local address 127.28.104.254:42435)
0619 06:39:17.227453 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:17.227471 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:17.227514 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:39:17.229232 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:17.229255 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:17.229669 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:17.229987 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:17.231677 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:17.232973 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:17.234955 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:17.235801 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:17.235933 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:17.240699 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:17.240714 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:17.240730 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:17.240774 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:17.243979 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:17.244662 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:17.244669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:17.244674 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:17.244883 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:17.245315 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:17.245320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:17.245324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:17.245846 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:17.246143 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:17.246653 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 06:39:17.247083 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":371,"threads_started":1}
I20260619 06:39:17.251703 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:17.252849 18880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707690505f2b44a79f6b99a78df323f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42435 } }
I20260619 06:39:17.253274 18880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:17.253377 18880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707690505f2b44a79f6b99a78df323f5, State: Initialized, Role: FOLLOWER
I20260619 06:39:17.253917 18880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707690505f2b44a79f6b99a78df323f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42435 } }
I20260619 06:39:17.254180 18880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:17.254325 18880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:17.254525 18880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:17.257539 18880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707690505f2b44a79f6b99a78df323f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42435 } }
I20260619 06:39:17.258034 18880 leader_election.cc:302] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 707690505f2b44a79f6b99a78df323f5; no voters:
I20260619 06:39:17.258699 18880 leader_election.cc:288] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:17.258805 18890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:17.261379 18880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:17.265944 18890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [term 1 LEADER]: Becoming Leader. State: Replica: 707690505f2b44a79f6b99a78df323f5, State: Running, Role: LEADER
I20260619 06:39:17.266769 18890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707690505f2b44a79f6b99a78df323f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42435 } }
I20260619 06:39:17.270134 18892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 707690505f2b44a79f6b99a78df323f5. Latest consensus state: current_term: 1 leader_uuid: "707690505f2b44a79f6b99a78df323f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707690505f2b44a79f6b99a78df323f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42435 } } }
I20260619 06:39:17.270582 18892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:17.271552 18901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:17.271407 18891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "707690505f2b44a79f6b99a78df323f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707690505f2b44a79f6b99a78df323f5" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 42435 } } }
I20260619 06:39:17.271878 18891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:17.271910 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:17.256162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49845 (local address 127.28.104.254:42435)
0619 06:39:17.256357 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:17.256365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:17.256745 (+ 380us) server_negotiation.cc:407] Connection header received
0619 06:39:17.256939 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:17.256944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:17.257011 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:17.257139 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:17.258582 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:17.259750 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:17.261072 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:17.261568 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:17.261694 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:17.264303 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:17.264308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:17.264311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:17.264340 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:17.267809 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:17.268480 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:17.268485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:17.268488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:17.268672 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:17.269180 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:17.269190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:17.269194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:17.269618 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:17.269857 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:17.270984 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0619 06:39:17.271297 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:39:17.280222 18901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:17.293396 18901 catalog_manager.cc:1347] Generated new cluster ID: a2ffe0aa7d094b299a98a0942986fee7
I20260619 06:39:17.293509 18901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:17.318080 18901 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:17.320709 18901 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:17.331017 18901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 707690505f2b44a79f6b99a78df323f5: Generated new TSK 0
I20260619 06:39:17.332204 18901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:17.339977 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18813
I20260619 06:39:17.345795 18875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:17.442333 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 18813
2026-06-19T06:39:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2052 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:17 dist-test-slave-nf4n krb5kdc[18913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:17 dist-test-slave-nf4n krb5kdc[18913](info): set up 2 sockets
Jun 19 06:39:17 dist-test-slave-nf4n krb5kdc[18913](info): commencing operation
krb5kdc: starting...
W20260619 06:39:18.545275 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:18 dist-test-slave-nf4n krb5kdc[18913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:39:18.793941 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:39191
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:37855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:39191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:19.119426 18926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:19.119807 18926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:19.119982 18926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:19.127506 18926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:19.127673 18926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:19.127794 18926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:19.127911 18926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:19.127998 18926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:19.137913 18926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:37855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:39191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:39191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:19.139253 18926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:19.140707 18926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:19.152796 18931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:19.152920 18932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:19.152863 18934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:19.154063 18926 server_base.cc:1034] running on GCE node
I20260619 06:39:19.154794 18926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:19.156111 18926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:19.157367 18926 hybrid_clock.cc:648] HybridClock initialized: now 1781851159157344 us; error 43 us; skew 500 ppm
I20260619 06:39:19.157759 18926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:19 dist-test-slave-nf4n krb5kdc[18913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851159, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:19.161156 18926 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:39:19.162618 18926 webserver.cc:460] Webserver started at http://127.28.104.254:46527/ using document root <none> and password file <none>
I20260619 06:39:19.163360 18926 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:19.163520 18926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:19.163851 18926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:19.166980 18926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "1f074b6a308d4e06b33c6cba4e21d618"
format_stamp: "Formatted at 2026-06-19 06:39:19 on dist-test-slave-nf4n"
server_key: "e5923c4daad2a09412fdfdecb462f85e"
server_key_iv: "b76532d9ad4ec3780922aaa2a7a73c46"
server_key_version: "encryptionkey@0"
I20260619 06:39:19.167824 18926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "1f074b6a308d4e06b33c6cba4e21d618"
format_stamp: "Formatted at 2026-06-19 06:39:19 on dist-test-slave-nf4n"
server_key: "e5923c4daad2a09412fdfdecb462f85e"
server_key_iv: "b76532d9ad4ec3780922aaa2a7a73c46"
server_key_version: "encryptionkey@0"
I20260619 06:39:19.173292 18926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:39:19.176805 18941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:19.178375 18926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:39:19.178602 18926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "1f074b6a308d4e06b33c6cba4e21d618"
format_stamp: "Formatted at 2026-06-19 06:39:19 on dist-test-slave-nf4n"
server_key: "e5923c4daad2a09412fdfdecb462f85e"
server_key_iv: "b76532d9ad4ec3780922aaa2a7a73c46"
server_key_version: "encryptionkey@0"
I20260619 06:39:19.178840 18926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:19.198243 18926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:19.203359 18926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:19.203692 18926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:19.223901 18926 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:39191
I20260619 06:39:19.223907 18993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:39191 every 8 connection(s)
I20260619 06:39:19.225456 18926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:19.228165 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 18926
I20260619 06:39:19.228344 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:19.228891 29091 external_mini_cluster.cc:1428] Setting key cfb8166780f88abe38d7d7c69e48d274
I20260619 06:39:19.229930 18994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:19.243754 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:19.230729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57819 (local address 127.28.104.254:39191)
0619 06:39:19.231507 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:19.231522 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:19.231572 (+ 50us) server_negotiation.cc:407] Connection header received
0619 06:39:19.233563 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:19.233590 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:19.234117 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:19.234545 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:19.235053 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:19.237092 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:19.238379 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:19.239400 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:19.239578 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:19.240928 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:19.240942 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:19.240960 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:19.241742 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:19.241839 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:19.241875 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:19.241989 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:19.242229 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 06:39:19.242505 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":332,"threads_started":1}
I20260619 06:39:19.245007 18994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618: Bootstrap starting.
I20260619 06:39:19.247529 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:19.248437 18994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:19.249658 18994 log.cc:824] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:19.252617 18994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618: No bootstrap required, opened a new log
I20260619 06:39:19.259166 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:19.252611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41429 (local address 127.28.104.254:39191)
0619 06:39:19.252790 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:19.252794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:19.253350 (+ 556us) server_negotiation.cc:407] Connection header received
0619 06:39:19.253502 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:19.253508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:19.253587 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:19.253692 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:19.254142 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:19.255344 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:19.256604 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:19.257300 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:19.257441 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:19.258416 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:19.258422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:19.258427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:19.258448 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:19.258501 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:19.258519 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:19.258626 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:19.258722 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 06:39:19.258820 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:39:19.262054 18994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f074b6a308d4e06b33c6cba4e21d618" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39191 } }
I20260619 06:39:19.262418 18994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:19.262542 18994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f074b6a308d4e06b33c6cba4e21d618, State: Initialized, Role: FOLLOWER
I20260619 06:39:19.263091 18994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f074b6a308d4e06b33c6cba4e21d618" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39191 } }
I20260619 06:39:19.263321 18994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:19.263478 18994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:19.263697 18994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:19.265898 18994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f074b6a308d4e06b33c6cba4e21d618" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39191 } }
I20260619 06:39:19.266388 18994 leader_election.cc:302] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f074b6a308d4e06b33c6cba4e21d618; no voters:
I20260619 06:39:19.266921 18994 leader_election.cc:288] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:19.267066 19004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:19.267823 19004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [term 1 LEADER]: Becoming Leader. State: Replica: 1f074b6a308d4e06b33c6cba4e21d618, State: Running, Role: LEADER
I20260619 06:39:19.268373 19004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f074b6a308d4e06b33c6cba4e21d618" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39191 } }
I20260619 06:39:19.269158 18994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:19.270977 19006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f074b6a308d4e06b33c6cba4e21d618. Latest consensus state: current_term: 1 leader_uuid: "1f074b6a308d4e06b33c6cba4e21d618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f074b6a308d4e06b33c6cba4e21d618" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39191 } } }
I20260619 06:39:19.271340 19006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:19.272069 19005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f074b6a308d4e06b33c6cba4e21d618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f074b6a308d4e06b33c6cba4e21d618" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 39191 } } }
I20260619 06:39:19.272470 19005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:19.273825 19009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:19.280171 19009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:19.289422 19009 catalog_manager.cc:1347] Generated new cluster ID: 124c2d65f3d140a5997a9157f9fbbf03
I20260619 06:39:19.289531 19009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:19.301112 19009 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:19.302461 19009 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:19.309150 19009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f074b6a308d4e06b33c6cba4e21d618: Generated new TSK 0
I20260619 06:39:19.309953 19009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:19.317921 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18926
W20260619 06:39:19.318941 18989 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57819
W20260619 06:39:19.319610 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 18926: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:19.319767 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 18926
2026-06-19T06:39:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:19 dist-test-slave-nf4n krb5kdc[19025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:19 dist-test-slave-nf4n krb5kdc[19025](info): set up 2 sockets
Jun 19 06:39:19 dist-test-slave-nf4n krb5kdc[19025](info): commencing operation
krb5kdc: starting...
W20260619 06:39:20.429257 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:20 dist-test-slave-nf4n krb5kdc[19025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:39:20.675860 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:46615
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:46615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:21.002305 19038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:21.002656 19038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:21.002820 19038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:21.010049 19038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:21.010210 19038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:21.010329 19038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:21.010440 19038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:21.010519 19038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:21.020562 19038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:46615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:21.021952 19038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:21.023262 19038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:21.035526 19043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:21.035497 19046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:21.035662 19044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:21.036291 19038 server_base.cc:1034] running on GCE node
I20260619 06:39:21.037092 19038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:21.038576 19038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:21.039772 19038 hybrid_clock.cc:648] HybridClock initialized: now 1781851161039714 us; error 48 us; skew 500 ppm
I20260619 06:39:21.040099 19038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:21 dist-test-slave-nf4n krb5kdc[19025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851161, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:21.047672 19038 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:39:21.049243 19038 webserver.cc:460] Webserver started at http://127.28.104.254:41789/ using document root <none> and password file <none>
I20260619 06:39:21.049916 19038 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:21.050056 19038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:21.050382 19038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:21.053375 19038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "676723668b02426aab7aee71a890fedd"
format_stamp: "Formatted at 2026-06-19 06:39:21 on dist-test-slave-nf4n"
server_key: "8ddcffc04042b0b9ab3e24b729b3d3dd"
server_key_iv: "82b6936aae964ec4043be45a20b3f749"
server_key_version: "encryptionkey@0"
I20260619 06:39:21.054126 19038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "676723668b02426aab7aee71a890fedd"
format_stamp: "Formatted at 2026-06-19 06:39:21 on dist-test-slave-nf4n"
server_key: "8ddcffc04042b0b9ab3e24b729b3d3dd"
server_key_iv: "82b6936aae964ec4043be45a20b3f749"
server_key_version: "encryptionkey@0"
I20260619 06:39:21.059129 19038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 06:39:21.062428 19053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:21.063644 19038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:39:21.063827 19038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "676723668b02426aab7aee71a890fedd"
format_stamp: "Formatted at 2026-06-19 06:39:21 on dist-test-slave-nf4n"
server_key: "8ddcffc04042b0b9ab3e24b729b3d3dd"
server_key_iv: "82b6936aae964ec4043be45a20b3f749"
server_key_version: "encryptionkey@0"
I20260619 06:39:21.064024 19038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:21.069927 19038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:21.073637 19038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:21.073923 19038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:21.091900 19038 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:46615
I20260619 06:39:21.091934 19104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:46615 every 8 connection(s)
I20260619 06:39:21.093362 19038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:21.096906 19106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:21.099217 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 19038
I20260619 06:39:21.099403 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:21.099844 29091 external_mini_cluster.cc:1428] Setting key a7f6d5ea6a689a9381140e9d0399f9f7
I20260619 06:39:21.108642 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:21.101654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45337 (local address 127.28.104.254:46615)
0619 06:39:21.102462 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:21.102477 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:21.102530 (+ 53us) server_negotiation.cc:407] Connection header received
0619 06:39:21.104849 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:21.104875 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:21.105324 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:21.105782 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:21.106737 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:21.106746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:21.106768 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:21.106784 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:21.106851 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:21.106881 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:21.107102 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:21.107251 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 06:39:21.107437 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":332,"threads_started":1}
I20260619 06:39:21.110388 19106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd: Bootstrap starting.
I20260619 06:39:21.112272 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:21.113756 19106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:21.114728 19106 log.cc:824] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:21.117722 19106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd: No bootstrap required, opened a new log
I20260619 06:39:21.119176 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:21.116697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39261 (local address 127.28.104.254:46615)
0619 06:39:21.116903 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:21.116911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:21.117250 (+ 339us) server_negotiation.cc:407] Connection header received
0619 06:39:21.117449 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:21.117453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:21.117496 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:21.117591 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:21.118512 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:21.118517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:21.118521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:21.118538 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:21.118583 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:21.118598 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:21.118717 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:21.118822 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 06:39:21.118894 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:39:21.125751 19106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676723668b02426aab7aee71a890fedd" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 46615 } }
I20260619 06:39:21.126142 19106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:21.126266 19106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676723668b02426aab7aee71a890fedd, State: Initialized, Role: FOLLOWER
I20260619 06:39:21.126868 19106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676723668b02426aab7aee71a890fedd" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 46615 } }
I20260619 06:39:21.127175 19106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:21.127336 19106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:21.127527 19106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:21.129902 19106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676723668b02426aab7aee71a890fedd" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 46615 } }
I20260619 06:39:21.130327 19106 leader_election.cc:302] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 676723668b02426aab7aee71a890fedd; no voters:
I20260619 06:39:21.130818 19106 leader_election.cc:288] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:21.130966 19116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:21.132062 19116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [term 1 LEADER]: Becoming Leader. State: Replica: 676723668b02426aab7aee71a890fedd, State: Running, Role: LEADER
I20260619 06:39:21.132581 19116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676723668b02426aab7aee71a890fedd" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 46615 } }
I20260619 06:39:21.132978 19106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:21.135522 19117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "676723668b02426aab7aee71a890fedd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676723668b02426aab7aee71a890fedd" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 46615 } } }
I20260619 06:39:21.135689 19118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 676723668b02426aab7aee71a890fedd. Latest consensus state: current_term: 1 leader_uuid: "676723668b02426aab7aee71a890fedd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676723668b02426aab7aee71a890fedd" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 46615 } } }
I20260619 06:39:21.135941 19117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:21.135941 19118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:21.137609 19125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:21.142027 19125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:21.150712 19125 catalog_manager.cc:1347] Generated new cluster ID: b0f92351d0e34a819b99bd837b6f9e2e
I20260619 06:39:21.150815 19125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:21.170267 19125 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:21.170715 19125 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:21.178236 19125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 676723668b02426aab7aee71a890fedd: Generated new TSK 0
I20260619 06:39:21.179019 19125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:21.191620 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 19038
W20260619 06:39:21.192652 19100 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45337
W20260619 06:39:21.193379 29091 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 19038: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:21.193529 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 19038
2026-06-19T06:39:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1874 ms)
[----------] 18 tests from AuthTokenIssuingTest (34878 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:21 dist-test-slave-nf4n krb5kdc[19137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:21 dist-test-slave-nf4n krb5kdc[19137](info): set up 2 sockets
Jun 19 06:39:21 dist-test-slave-nf4n krb5kdc[19137](info): commencing operation
krb5kdc: starting...
W20260619 06:39:22.305392 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:22 dist-test-slave-nf4n krb5kdc[19137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:39:22.566637 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:38515
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:38515,127.28.104.253:36199,127.28.104.252:34833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:22.891990 19150 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:39:22.892460 19150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:22.892570 19150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:22.892714 19150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:22.900112 19150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:22.900203 19150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:22.900310 19150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:22.900398 19150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:22.900489 19150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:22.910177 19150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:38515,127.28.104.253:36199,127.28.104.252:34833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:22.911471 19150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:22.912868 19150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:22.924645 19155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:22.924549 19156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:22.924584 19158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:22.925345 19150 server_base.cc:1034] running on GCE node
I20260619 06:39:22.926156 19150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:22.927498 19150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:22.928730 19150 hybrid_clock.cc:648] HybridClock initialized: now 1781851162928664 us; error 56 us; skew 500 ppm
I20260619 06:39:22.929066 19150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:22 dist-test-slave-nf4n krb5kdc[19137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851162, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:22.932209 19150 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:39:22.933805 19150 webserver.cc:460] Webserver started at http://127.28.104.254:38851/ using document root <none> and password file <none>
I20260619 06:39:22.934489 19150 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:22.934612 19150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:22.934962 19150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:22.937984 19150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "b1b38df194d245eaa95c5c9fe8300246"
format_stamp: "Formatted at 2026-06-19 06:39:22 on dist-test-slave-nf4n"
server_key: "1a03dfa446d77922771338bcf58bc312"
server_key_iv: "c73a8b03b9a133cdc94608dfcc0d6e47"
server_key_version: "encryptionkey@0"
I20260619 06:39:22.938818 19150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "b1b38df194d245eaa95c5c9fe8300246"
format_stamp: "Formatted at 2026-06-19 06:39:22 on dist-test-slave-nf4n"
server_key: "1a03dfa446d77922771338bcf58bc312"
server_key_iv: "c73a8b03b9a133cdc94608dfcc0d6e47"
server_key_version: "encryptionkey@0"
I20260619 06:39:22.944000 19150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:39:22.947516 19165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:22.948961 19150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:39:22.949208 19150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "b1b38df194d245eaa95c5c9fe8300246"
format_stamp: "Formatted at 2026-06-19 06:39:22 on dist-test-slave-nf4n"
server_key: "1a03dfa446d77922771338bcf58bc312"
server_key_iv: "c73a8b03b9a133cdc94608dfcc0d6e47"
server_key_version: "encryptionkey@0"
I20260619 06:39:22.949437 19150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:22.983834 19150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:22.988452 19150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:22.988780 19150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:23.009234 19150 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:38515
I20260619 06:39:23.009208 19216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:38515 every 8 connection(s)
I20260619 06:39:23.010787 19150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:23.011698 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 19150
I20260619 06:39:23.011937 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:23.012709 29091 external_mini_cluster.cc:1428] Setting key 3029f58e6cfd53085d391296dfa1e938
I20260619 06:39:23.015298 19217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:23.022847 19217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.104.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.253@KRBTEST.COM" created.
Jun 19 06:39:23 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851162, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:39:23.052521 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.027933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.28.104.254:38515)
0619 06:39:23.028549 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:23.028557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:23.028600 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:39:23.031211 (+ 2611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:23.031222 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:23.031472 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:23.031631 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:23.033478 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.034704 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.037152 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.037730 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.037858 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:23.044127 (+ 6269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:23.044141 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:23.044161 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:23.044219 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:23.048150 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:23.048863 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:23.048874 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:23.048880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:23.049148 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:23.049552 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:23.049561 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:23.049567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:23.050287 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:23.050491 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:23.051111 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 06:39:23.051578 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":457,"thread_start_us":303,"threads_started":1}
I20260619 06:39:23.052618 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.026877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:38515 (local address 127.0.0.1:55532)
0619 06:39:23.027728 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.027788 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:23.030773 (+ 2985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:23.031751 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:23.031766 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:23.032623 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:23.033200 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:23.033232 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.034885 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.034895 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:23.036960 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:23.036980 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.037989 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.038000 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:23.038275 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:23.039067 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:23.039110 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:23.043857 (+ 4747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:23.048293 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:23.048302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:23.048312 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:23.048713 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:23.049276 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:23.049279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:23.049282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:23.049432 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:23.050462 (+ 1030us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:23.050468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:23.050694 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 06:39:23.050925 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 06:39:23.051356 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":325,"threads_started":1}
I20260619 06:39:23.056214 19217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:39:23.057392 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.056972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50098)
0619 06:39:23.057187 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.057287 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260619 06:39:23.057806 19166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
I20260619 06:39:23.059353 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.059023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50108)
0619 06:39:23.059182 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.059254 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:39:23.059690 19217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:36199: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab.
Entry for principal kudu/127.28.104.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab.
I20260619 06:39:23.076059 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.253:36199
--webserver_interface=127.28.104.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab
--principal=kudu/127.28.104.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:38515,127.28.104.253:36199,127.28.104.252:34833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:39:23.109108 19217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } attempt: 1
I20260619 06:39:23.110494 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.109860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50112)
0619 06:39:23.110238 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.110349 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
I20260619 06:39:23.111526 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.111173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50128)
0619 06:39:23.111337 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.111403 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:39:23.111706 19217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:36199: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
I20260619 06:39:23.180027 19217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } attempt: 2
I20260619 06:39:23.181285 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.180776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50138)
0619 06:39:23.181009 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.181142 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 06:39:23.182279 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.181931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50154)
0619 06:39:23.182085 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.182158 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 06:39:23.182447 19217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:36199: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
I20260619 06:39:23.273694 19217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } attempt: 3
I20260619 06:39:23.274806 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.274365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50160)
0619 06:39:23.274575 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.274672 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 06:39:23.275830 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.275486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50174)
0619 06:39:23.275638 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.275702 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:39:23.276007 19217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:36199: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
W20260619 06:39:23.406158 19224 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:39:23.406579 19224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:23.406724 19224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:23.406876 19224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:23.414259 19224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:23.414400 19224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:23.414516 19224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:23.414618 19224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:23.414736 19224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:23.419232 19217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } attempt: 4
I20260619 06:39:23.420468 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.419895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50186)
0619 06:39:23.420242 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.420331 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260619 06:39:23.421550 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.421229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50188)
0619 06:39:23.421377 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.421438 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:39:23.421729 19217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:36199: Network error: Client connection negotiation failed: client connection to 127.28.104.253:36199: connect: Connection refused (error 111)
I20260619 06:39:23.425536 19224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:38515,127.28.104.253:36199,127.28.104.252:34833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab
--principal=kudu/127.28.104.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.253:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.104.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:23.426890 19224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:23.428263 19224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:23.440539 19232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:23.440583 19229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:23.440498 19230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:23.441638 19224 server_base.cc:1034] running on GCE node
I20260619 06:39:23.442428 19224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:23.443794 19224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:23.445093 19224 hybrid_clock.cc:648] HybridClock initialized: now 1781851163444954 us; error 65 us; skew 500 ppm
I20260619 06:39:23.445444 19224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:23 dist-test-slave-nf4n krb5kdc[19137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:23.448607 19224 init.cc:377] Logged in from keytab as kudu/127.28.104.253@KRBTEST.COM (short username kudu)
I20260619 06:39:23.450098 19224 webserver.cc:460] Webserver started at http://127.28.104.253:36863/ using document root <none> and password file <none>
I20260619 06:39:23.450963 19224 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:23.451140 19224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:23.451521 19224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:23.455572 19224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/data/instance:
uuid: "9ba420c7beb24f33adf0568c6f84b418"
format_stamp: "Formatted at 2026-06-19 06:39:23 on dist-test-slave-nf4n"
server_key: "54f2424c763754b538c6388737c9b464"
server_key_iv: "352f44cd6088ac0d475bd39011a13a86"
server_key_version: "encryptionkey@0"
I20260619 06:39:23.456384 19224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/wal/instance:
uuid: "9ba420c7beb24f33adf0568c6f84b418"
format_stamp: "Formatted at 2026-06-19 06:39:23 on dist-test-slave-nf4n"
server_key: "54f2424c763754b538c6388737c9b464"
server_key_iv: "352f44cd6088ac0d475bd39011a13a86"
server_key_version: "encryptionkey@0"
I20260619 06:39:23.461541 19224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 06:39:23.465095 19239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:23.466470 19224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:23.466653 19224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/wal
uuid: "9ba420c7beb24f33adf0568c6f84b418"
format_stamp: "Formatted at 2026-06-19 06:39:23 on dist-test-slave-nf4n"
server_key: "54f2424c763754b538c6388737c9b464"
server_key_iv: "352f44cd6088ac0d475bd39011a13a86"
server_key_version: "encryptionkey@0"
I20260619 06:39:23.466873 19224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:23.480218 19224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:23.484486 19224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:23.484761 19224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:23.503382 19224 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.253:36199
I20260619 06:39:23.503391 19291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.253:36199 every 8 connection(s)
I20260619 06:39:23.504724 19224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
I20260619 06:39:23.508253 19292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:23.510459 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 19224
I20260619 06:39:23.510637 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-1/wal/instance
I20260619 06:39:23.511168 29091 external_mini_cluster.cc:1428] Setting key 7ed868665c1d7e9f12ec12ad1de39e4e
I20260619 06:39:23.517278 19292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:39:23 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.104.252@KRBTEST.COM; defaulting to no policy
I20260619 06:39:23.540269 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.520547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.28.104.254:38515)
0619 06:39:23.520778 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:23.520786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:23.521956 (+ 1170us) server_negotiation.cc:407] Connection header received
0619 06:39:23.524654 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:23.524658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:23.524731 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:23.524824 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:23.526876 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.527523 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.529908 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.530307 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.530407 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:23.534536 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:23.534541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:23.534545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:23.534575 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:23.537272 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:23.537978 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:23.537989 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:23.537991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:23.538108 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:23.538522 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:23.538527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:23.538530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:23.538834 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:23.539011 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:23.539663 (+ 652us) server_negotiation.cc:299] Negotiation successful
0619 06:39:23.539949 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.252@KRBTEST.COM" created.
I20260619 06:39:23.541393 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.520961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:38515 (local address 127.0.0.1:55536)
0619 06:39:23.521782 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.521831 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:23.524365 (+ 2534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:23.525121 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:23.525135 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:23.525893 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:23.526695 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:23.526721 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.527681 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.527689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:23.529774 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:23.529790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.530523 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.530532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:23.530845 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:23.531597 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:23.531630 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:23.534240 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:23.537408 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:23.537417 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:23.537433 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:23.537852 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:23.538225 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:23.538230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:23.538234 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:23.538406 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:23.538975 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:23.538982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:23.539316 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 06:39:23.539569 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 06:39:23.540055 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":334,"threads_started":1}
I20260619 06:39:23.543175 19292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:39:23 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
I20260619 06:39:23.557987 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.543988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50196)
0619 06:39:23.544173 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.544193 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:23.544284 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:23.545301 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:23.545305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:23.545726 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:23.545846 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:23.545854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.547200 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.547208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:23.548039 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:23.548049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.548854 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.548859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:23.549099 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:23.549671 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:23.549687 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:23.551856 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:23.555542 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:23.555547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:23.555551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:23.555926 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:23.556298 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:23.556303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:23.556306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:23.556427 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:23.556994 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:23.556998 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:23.557185 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 06:39:23.557328 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 06:39:23.557597 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 06:39:23.558092 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.544259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.28.104.253:36199)
0619 06:39:23.544855 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:23.544860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:23.544884 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:39:23.544926 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:23.544931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:23.545165 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:23.545318 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:23.546008 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.547066 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.548187 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.548715 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.548846 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:23.552083 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:23.552099 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:23.552119 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:23.552162 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:23.555410 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:23.556054 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:23.556064 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:23.556072 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:23.556180 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:23.556534 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:23.556538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:23.556542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:23.556873 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:23.557113 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:23.557485 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 06:39:23.557721 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":318,"threads_started":1}
I20260619 06:39:23.560349 19292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:39:23.561539 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.561105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:51954)
0619 06:39:23.561283 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.561405 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:39:23.561914 19242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
I20260619 06:39:23.563525 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.563187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:51966)
0619 06:39:23.563360 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.563430 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:39:23.563845 19292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:34833: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab.
Entry for principal kudu/127.28.104.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab.
I20260619 06:39:23.571700 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.252:34833
--webserver_interface=127.28.104.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab
--principal=kudu/127.28.104.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:38515,127.28.104.253:36199,127.28.104.252:34833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:39:23.613283 19292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } attempt: 1
I20260619 06:39:23.614699 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.614210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:51970)
0619 06:39:23.614387 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.614538 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 06:39:23.615767 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.615428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:51978)
0619 06:39:23.615586 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.615667 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:39:23.616051 19292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:34833: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
I20260619 06:39:23.684306 19292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } attempt: 2
I20260619 06:39:23.685503 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.685069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:51988)
0619 06:39:23.685259 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.685351 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 06:39:23.686450 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.686153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:51996)
0619 06:39:23.686274 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.686338 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:39:23.686692 19292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:34833: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
I20260619 06:39:23.720973 19217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } attempt: 5
Jun 19 06:39:23 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851162, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
I20260619 06:39:23.736302 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.721770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50198)
0619 06:39:23.721981 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.722000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:23.722119 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:23.722504 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:23.722507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:23.722882 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:23.723041 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:23.723049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.724611 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.724615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:23.725588 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:23.725602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.726538 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.726542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:23.726696 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:23.727332 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:23.727349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:23.729740 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:23.733926 (+ 4186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:23.733930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:23.733934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:23.734296 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:23.734761 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:23.734764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:23.734767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:23.734980 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:23.735578 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:23.735581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:23.735726 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 06:39:23.735836 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 06:39:23.736037 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 06:39:23.736490 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.721893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.28.104.253:36199)
0619 06:39:23.722078 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:23.722084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:23.722108 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:39:23.722253 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:23.722258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:23.722355 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:23.722469 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:23.723216 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.724463 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.725755 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:23.726399 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:23.726533 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:23.729958 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:23.729967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:23.729973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:23.730024 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:23.733775 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:23.734413 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:23.734420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:23.734426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:23.734577 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:23.735093 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:23.735098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:23.735101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:23.735463 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:23.735613 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:23.735965 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 06:39:23.736187 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:39:23.737576 19217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:39:23.738605 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.738258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52000)
0619 06:39:23.738413 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.738485 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:39:23.739519 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.739204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52012)
0619 06:39:23.739357 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.739424 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:39:23.739738 19217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:34833: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
I20260619 06:39:23.777909 19292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } attempt: 3
I20260619 06:39:23.779134 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.778687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52026)
0619 06:39:23.778888 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.778987 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 06:39:23.780195 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.779780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52032)
0619 06:39:23.779995 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.780072 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260619 06:39:23.780392 19292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:34833: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
I20260619 06:39:23.790930 19217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } attempt: 1
I20260619 06:39:23.791945 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.791530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52046)
0619 06:39:23.791719 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.791808 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 06:39:23.792883 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.792552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52050)
0619 06:39:23.792714 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.792779 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:39:23.793093 19217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:34833: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
I20260619 06:39:23.861303 19217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } attempt: 2
I20260619 06:39:23.862422 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.861933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52052)
0619 06:39:23.862163 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.862253 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 06:39:23.863413 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.863093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52056)
0619 06:39:23.863238 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.863324 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:39:23.863605 19217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:34833: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
W20260619 06:39:23.901367 19299 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:39:23.901726 19299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:23.901846 19299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:23.902009 19299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:23.909163 19299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:23.909279 19299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:23.909379 19299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:23.909462 19299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:23.909556 19299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:23.919201 19299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:38515,127.28.104.253:36199,127.28.104.252:34833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab
--principal=kudu/127.28.104.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.252:34833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.104.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:23.920475 19299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:23.921790 19299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:39:23.923578 19292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } attempt: 4
I20260619 06:39:23.924670 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.924199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52062)
0619 06:39:23.924425 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.924509 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 06:39:23.925664 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.925346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52076)
0619 06:39:23.925488 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.925561 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:39:23.925877 19292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:34833: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
W20260619 06:39:23.933895 19305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:23.933936 19304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:23.933972 19307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:23.935370 19299 server_base.cc:1034] running on GCE node
I20260619 06:39:23.936131 19299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:23.937657 19299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:23.938896 19299 hybrid_clock.cc:648] HybridClock initialized: now 1781851163938830 us; error 47 us; skew 500 ppm
I20260619 06:39:23.939234 19299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:23 dist-test-slave-nf4n krb5kdc[19137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:23.942384 19299 init.cc:377] Logged in from keytab as kudu/127.28.104.252@KRBTEST.COM (short username kudu)
I20260619 06:39:23.943863 19299 webserver.cc:460] Webserver started at http://127.28.104.252:43729/ using document root <none> and password file <none>
I20260619 06:39:23.944514 19299 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:23.944598 19299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:23.944866 19299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:23.947882 19299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/data/instance:
uuid: "cd92679848d84134b1d7338c6f392549"
format_stamp: "Formatted at 2026-06-19 06:39:23 on dist-test-slave-nf4n"
server_key: "2d93cf5f9edb47ba43a5b39caf96d9d6"
server_key_iv: "196af833f085700647b800d32ef24f37"
server_key_version: "encryptionkey@0"
I20260619 06:39:23.948606 19299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/wal/instance:
uuid: "cd92679848d84134b1d7338c6f392549"
format_stamp: "Formatted at 2026-06-19 06:39:23 on dist-test-slave-nf4n"
server_key: "2d93cf5f9edb47ba43a5b39caf96d9d6"
server_key_iv: "196af833f085700647b800d32ef24f37"
server_key_version: "encryptionkey@0"
I20260619 06:39:23.953665 19299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:39:23.957072 19314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:23.958369 19299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:23.958523 19299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/wal
uuid: "cd92679848d84134b1d7338c6f392549"
format_stamp: "Formatted at 2026-06-19 06:39:23 on dist-test-slave-nf4n"
server_key: "2d93cf5f9edb47ba43a5b39caf96d9d6"
server_key_iv: "196af833f085700647b800d32ef24f37"
server_key_version: "encryptionkey@0"
I20260619 06:39:23.958710 19299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:23.969815 19217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } attempt: 3
I20260619 06:39:23.970912 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.970456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52092)
0619 06:39:23.970679 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.970764 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:39:23.971992 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:23.971600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52104)
0619 06:39:23.971833 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:23.971896 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260619 06:39:23.972220 19217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:34833: Network error: Client connection negotiation failed: client connection to 127.28.104.252:34833: connect: Connection refused (error 111)
I20260619 06:39:23.980981 19299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:23.984894 19299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:23.985191 19299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:24.003894 19299 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.252:34833
I20260619 06:39:24.003912 19365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.252:34833 every 8 connection(s)
I20260619 06:39:24.005272 19299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
I20260619 06:39:24.009490 19366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:24.015118 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 19299
I20260619 06:39:24.015339 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850805520649-29091-0/minicluster-data/master-2/wal/instance
I20260619 06:39:24.015763 29091 external_mini_cluster.cc:1428] Setting key 07b9e575b4f16d90698f99b685bcf3fc
I20260619 06:39:24.018607 19366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:39:24 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
Jun 19 06:39:24 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
I20260619 06:39:24.037923 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.022533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.28.104.254:38515)
0619 06:39:24.022764 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.022774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.023409 (+ 635us) server_negotiation.cc:407] Connection header received
0619 06:39:24.023641 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.023649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.023723 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.023829 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.024986 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.025742 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.027103 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.027526 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.027652 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.031057 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.031062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.031066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.031097 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.034414 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.035396 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.035400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.035402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.035581 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.036059 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.036065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.036070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.036422 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.036556 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.037277 (+ 721us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.037532 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:39:24.038848 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.017583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.28.104.252:34833)
0619 06:39:24.018437 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.018453 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.018501 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:39:24.020731 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.020754 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.021254 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.021626 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.023297 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.024842 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.026158 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.026800 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.027019 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.031625 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.031639 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.031660 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.031693 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.035175 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.036046 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.036054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.036057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.036199 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.036599 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.036603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.036605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.036940 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.037301 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.037580 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.037817 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":390,"threads_started":1}
I20260619 06:39:24.038898 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.022544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:38515 (local address 127.0.0.1:55540)
0619 06:39:24.023294 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:24.023328 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:24.023490 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:24.023897 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:24.023905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:24.024517 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:24.024824 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:24.024837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.025890 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.025895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:24.026957 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:24.026968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.027754 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.027762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:24.027917 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.028502 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:24.028526 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:24.030881 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:24.034541 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:24.034550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:24.034558 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:24.035282 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:24.035707 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:24.035713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:24.035717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:24.035955 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:24.036556 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:24.036566 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:24.036883 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 06:39:24.037179 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 06:39:24.037688 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":344,"threads_started":1}
I20260619 06:39:24.041258 19366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:39:24 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
Jun 19 06:39:24 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:39:24.054041 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.041987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:36199 (local address 127.0.0.1:50214)
0619 06:39:24.042237 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:24.042254 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:24.042343 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:24.042660 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:24.042664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:24.043010 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:24.043150 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:24.043158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.044110 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.044118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:24.044986 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:24.044997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.045838 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.045843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:24.046036 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.046582 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:24.046599 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:24.048556 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:24.051642 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:24.051649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:24.051652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:24.051990 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:24.052376 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:24.052381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:24.052385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:24.052520 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:24.053060 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:24.053066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:24.053223 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 06:39:24.053427 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 06:39:24.053691 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260619 06:39:24.054073 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.042218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.28.104.253:36199)
0619 06:39:24.042391 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.042399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.042417 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:39:24.042480 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.042483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.042536 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.042640 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.043302 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.043979 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.045177 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.045699 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.045831 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.048742 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.048747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.048750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.048780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.051520 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.052102 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.052109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.052112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.052241 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.052625 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.052632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.052635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.052901 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.053088 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.053515 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.053726 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:39:24.054991 19366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:39:24.056535 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.041920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.28.104.254:38515)
0619 06:39:24.042070 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.042075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.042121 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:39:24.042311 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.042316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.042388 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.042487 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.043586 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.044328 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.046042 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.046459 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.046588 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.050761 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.050767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.050771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.050806 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.053445 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.054078 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.054085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.054088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.054338 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.054808 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.054815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.054819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.055082 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.055371 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.056174 (+ 803us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.056337 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:39:24 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
Jun 19 06:39:24 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
I20260619 06:39:24.069106 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.055765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52120 (local address 127.28.104.252:34833)
0619 06:39:24.055893 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.055899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.055955 (+ 56us) server_negotiation.cc:407] Connection header received
0619 06:39:24.056119 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.056123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.056184 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.056273 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.057003 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.057936 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.059235 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.059862 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.059950 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.063169 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.063177 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.063182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.063228 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.066454 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.067115 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.067120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.067123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.067259 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.067589 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.067595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.067598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.067933 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.068031 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.068435 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.068800 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":44}
I20260619 06:39:24.069135 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.055650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52120)
0619 06:39:24.055893 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:24.055911 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:24.056005 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:24.056286 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:24.056290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:24.056676 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:24.056837 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:24.056847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.058070 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.058075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:24.059088 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:24.059100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.059995 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.060001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:24.060218 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.060731 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:24.060752 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:24.062960 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:24.066577 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:24.066584 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:24.066588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:24.067007 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:24.067368 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:24.067371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:24.067373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:24.067487 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:24.068043 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:24.068048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:24.068202 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 06:39:24.068360 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 06:39:24.068627 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260619 06:39:24.074030 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.058175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.28.104.253:36199)
0619 06:39:24.058320 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.058326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.058343 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:39:24.058386 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.058391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.058452 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.058544 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.059756 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.060788 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.062055 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.062581 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.062681 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.067887 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.067891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.067894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.067920 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.071449 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.072136 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.072142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.072145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.072340 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.072689 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.072703 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.072708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.073007 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.073280 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.073513 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.073738 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:39:24.074815 19366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549: Bootstrap starting.
I20260619 06:39:24.078502 19366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:24.079059 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:24.079823 19366 log.cc:824] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:24.082672 19366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549: No bootstrap required, opened a new log
I20260619 06:39:24.082309 19192 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b1b38df194d245eaa95c5c9fe8300246"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55556
I20260619 06:39:24.089582 19366 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } }
I20260619 06:39:24.089911 19366 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd92679848d84134b1d7338c6f392549, State: Initialized, Role: FOLLOWER
I20260619 06:39:24.090340 19366 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } }
I20260619 06:39:24.091171 19373 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } } }
I20260619 06:39:24.091496 19373 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:24.092432 19366 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:39:24.101920 19384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:39:24.102041 19384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:39:24.099637 19299 master_runner.cc:186] Error getting master registration for 127.28.104.254:38515: OK, instance_id { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" instance_seqno: 1781851163003259 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:39:24.110700 19299 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:39:24.149379 19217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } attempt: 4
Jun 19 06:39:24 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851162, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
I20260619 06:39:24.166401 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.150202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.28.104.252:34833)
0619 06:39:24.150417 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.150423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.150513 (+ 90us) server_negotiation.cc:407] Connection header received
0619 06:39:24.150712 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.150717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.150780 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.150898 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.151840 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.153068 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.154924 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.155688 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.155827 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.160045 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.160051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.160057 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.160099 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.163369 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.164167 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.164172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.164176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.164321 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.164755 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.164759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.164763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.165223 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.165395 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.165799 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.166022 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:39:24.166404 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.150144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52128)
0619 06:39:24.150410 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:24.150433 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:24.150578 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:24.150920 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:24.150925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:24.151436 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:24.151682 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:24.151695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.153235 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.153241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:24.154746 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:24.154762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.156077 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.156083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:24.156295 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.156992 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:24.157018 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:24.159845 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:24.163518 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:24.163523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:24.163528 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:24.164045 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:24.164440 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:24.164445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:24.164449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:24.164612 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:24.165357 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:24.165363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:24.165529 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 06:39:24.165698 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 06:39:24.165967 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 06:39:24.172282 19217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246: Bootstrap starting.
I20260619 06:39:24.176229 19217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:24.177560 19217 log.cc:824] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:24.180490 19217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246: No bootstrap required, opened a new log
I20260619 06:39:24.188035 19217 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } }
I20260619 06:39:24.188457 19217 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b38df194d245eaa95c5c9fe8300246, State: Initialized, Role: FOLLOWER
I20260619 06:39:24.188884 19217 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } }
I20260619 06:39:24.189877 19387 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } } }
I20260619 06:39:24.190279 19387 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:24.191108 19217 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:24.191226 19192 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 06:39:24.191398 19192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:24.195833 19192 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } }
I20260619 06:39:24.198278 19192 leader_election.cc:288] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [CANDIDATE]: Term 1 election: Requested vote from peers 9ba420c7beb24f33adf0568c6f84b418 (127.28.104.253:36199), cd92679848d84134b1d7338c6f392549 (127.28.104.252:34833)
I20260619 06:39:24.199235 19341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1b38df194d245eaa95c5c9fe8300246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cd92679848d84134b1d7338c6f392549"
I20260619 06:39:24.199757 19341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:24.204121 19341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1b38df194d245eaa95c5c9fe8300246 in term 1.
I20260619 06:39:24.204870 19168 leader_election.cc:302] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1b38df194d245eaa95c5c9fe8300246, cd92679848d84134b1d7338c6f392549; no voters:
I20260619 06:39:24.205332 19387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:24.207441 19387 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [term 1 LEADER]: Becoming Leader. State: Replica: b1b38df194d245eaa95c5c9fe8300246, State: Running, Role: LEADER
I20260619 06:39:24.208004 19387 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } }
I20260619 06:39:24.221421 19394 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1b38df194d245eaa95c5c9fe8300246. Latest consensus state: current_term: 1 leader_uuid: "b1b38df194d245eaa95c5c9fe8300246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } } }
I20260619 06:39:24.221987 19394 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:24.225083 19292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } attempt: 5
I20260619 06:39:24.234709 19402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260619 06:39:24.237357 19320 tablet.cc:2369] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:39:24.241709 19171 tablet.cc:2369] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:39:24.248560 19402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:24.256172 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.218691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.28.104.254:38515)
0619 06:39:24.218896 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.218904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.228540 (+ 9636us) server_negotiation.cc:407] Connection header received
0619 06:39:24.229524 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.229533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.229631 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.229726 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.236342 (+ 6616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.237609 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.240978 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.242029 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.242222 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.246372 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.246380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.246385 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.246430 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.250572 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.252375 (+ 1803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.252381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.252384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.252602 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.253296 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.253305 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.253311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.253827 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.254075 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.255440 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.255699 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 06:39:24 dist-test-slave-nf4n krb5kdc[19137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
I20260619 06:39:24.262027 19410 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:39:24.277951 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.225728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:34833 (local address 127.0.0.1:52152)
0619 06:39:24.226094 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:24.226119 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:24.226250 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:24.250373 (+ 24123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:24.250377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:24.250884 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:24.251106 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:24.251119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.252957 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.252965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:24.254478 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:24.254492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.255630 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.255638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:24.255861 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.256490 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:24.256513 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:24.259077 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:24.275062 (+ 15985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:24.275067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:24.275072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:24.275589 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:24.276011 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:24.276016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:24.276020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:24.276231 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:24.276884 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:24.276889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:24.277078 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 06:39:24.277230 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 06:39:24.277530 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1}
I20260619 06:39:24.279119 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.226472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.28.104.252:34833)
0619 06:39:24.249651 (+ 23179us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.249657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.249682 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:39:24.249735 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.249744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.249817 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.251258 (+ 1441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.251509 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.252786 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.254670 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.255323 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.259538 (+ 4215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.259781 (+ 243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.259790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.259797 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.259833 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.273366 (+ 13533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.275747 (+ 2381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.275752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.275755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.275892 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.276377 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.276382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.276387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.276736 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.278350 (+ 1614us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.278477 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.278711 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23029,"thread_start_us":23704,"threads_started":1}
I20260619 06:39:24.280843 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.218502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.28.104.252:34833)
0619 06:39:24.218686 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.218691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.240996 (+ 22305us) server_negotiation.cc:407] Connection header received
0619 06:39:24.249429 (+ 8433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.249438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.249540 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.249637 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.253823 (+ 4186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.254848 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.262810 (+ 7962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.263514 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.263675 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.271414 (+ 7739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.271422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.271427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.271475 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.274854 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.276244 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.276252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.276257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.276525 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.277533 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.277539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.277542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.278209 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.278688 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.279771 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.280179 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:39:24.281951 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:24.218255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.28.104.253:36199)
0619 06:39:24.218452 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.218459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.249904 (+ 31445us) server_negotiation.cc:407] Connection header received
0619 06:39:24.250268 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.250274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.250358 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.250423 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.253545 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.254669 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.257910 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:24.258586 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:24.258728 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:24.261837 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:24.261845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:24.261849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:24.261888 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:24.265700 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.278361 (+ 12661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.278368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.278373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.278522 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:24.279558 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:24.279563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:24.279567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:24.279943 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:24.280170 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:24.281320 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0619 06:39:24.281565 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":64}
I20260619 06:39:24.284865 19292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418: Bootstrap starting.
I20260619 06:39:24.266404 19150 master_runner.cc:186] Error getting master registration for 127.28.104.253:36199: OK, instance_id { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" instance_seqno: 1781851163498223 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:39:24.286197 19150 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:39:24.287896 19341 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [term 1 FOLLOWER]: Refusing update from remote peer b1b38df194d245eaa95c5c9fe8300246: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:24.288875 19394 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:24.292459 19292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:24.293797 19292 log.cc:824] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:24.317806 19292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418: No bootstrap required, opened a new log
I20260619 06:39:24.325465 19373 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1b38df194d245eaa95c5c9fe8300246. Latest consensus state: current_term: 1 leader_uuid: "b1b38df194d245eaa95c5c9fe8300246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } } }
I20260619 06:39:24.325778 19373 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:24.326941 19411 mvcc.cc:205] Tried to move back new op lower bound from 7298462368907849728 to 7298462368601989120. Current Snapshot: MvccSnapshot[applied={T|T < 7298462368907849728}]
I20260619 06:39:24.328824 19413 mvcc.cc:205] Tried to move back new op lower bound from 7298462368907849728 to 7298462368601989120. Current Snapshot: MvccSnapshot[applied={T|T < 7298462368907849728}]
I20260619 06:39:24.335041 19387 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1b38df194d245eaa95c5c9fe8300246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } } }
I20260619 06:39:24.335433 19387 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:24.340183 19373 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1b38df194d245eaa95c5c9fe8300246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } } }
I20260619 06:39:24.340432 19373 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:24.346777 19402 catalog_manager.cc:1347] Generated new cluster ID: 805b092af41e4c3c8327f98bcd6e32bb
I20260619 06:39:24.346905 19402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:24.362538 19292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } }
I20260619 06:39:24.363309 19292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba420c7beb24f33adf0568c6f84b418, State: Initialized, Role: FOLLOWER
I20260619 06:39:24.364102 19292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } }
I20260619 06:39:24.382339 19267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1b38df194d245eaa95c5c9fe8300246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ba420c7beb24f33adf0568c6f84b418"
I20260619 06:39:24.382896 19266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:24.384011 19292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:24.386889 19266 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [term 1 FOLLOWER]: Refusing update from remote peer b1b38df194d245eaa95c5c9fe8300246: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:24.388362 19402 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:24.390342 19402 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:24.392082 19267 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1b38df194d245eaa95c5c9fe8300246 in term 1.
I20260619 06:39:24.394750 19387 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:24.406446 19402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246: Generated new TSK 0
I20260619 06:39:24.407675 19402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:24.460352 19417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1b38df194d245eaa95c5c9fe8300246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } } }
I20260619 06:39:24.460906 19417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:24.460814 19425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1b38df194d245eaa95c5c9fe8300246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } } }
I20260619 06:39:24.461194 19425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:24.460402 19419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1b38df194d245eaa95c5c9fe8300246. Latest consensus state: current_term: 1 leader_uuid: "b1b38df194d245eaa95c5c9fe8300246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } } }
I20260619 06:39:24.461606 19419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:24.462299 19394 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1b38df194d245eaa95c5c9fe8300246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b38df194d245eaa95c5c9fe8300246" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 38515 } } peers { permanent_uuid: "9ba420c7beb24f33adf0568c6f84b418" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 36199 } } peers { permanent_uuid: "cd92679848d84134b1d7338c6f392549" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 34833 } } }
I20260619 06:39:24.462563 19394 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b38df194d245eaa95c5c9fe8300246 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:24.498086 19437 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 06:39:24.548264 19221 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.540361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.28.104.254:38515)
0619 06:39:24.540806 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.540814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.546073 (+ 5259us) server_negotiation.cc:407] Connection header received
0619 06:39:24.546559 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.546565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.546658 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.546739 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.547975 (+ 1236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48240: BlockingRecv error: recv got EOF from 127.0.0.1:48240 (error 108)
Metrics: {"server-negotiator.queue_time_us":270}
W20260619 06:39:24.550637 19403 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.539088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.28.104.252:34833)
0619 06:39:24.539996 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.540002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.549017 (+ 9015us) server_negotiation.cc:407] Connection header received
0619 06:39:24.549309 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.549315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.549417 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.549492 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.550392 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53558: BlockingRecv error: recv got EOF from 127.0.0.1:53558 (error 108)
Metrics: {"server-negotiator.queue_time_us":771}
W20260619 06:39:24.553124 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.540554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.28.104.253:36199)
0619 06:39:24.540735 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.540741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.551546 (+ 10805us) server_negotiation.cc:407] Connection header received
0619 06:39:24.551802 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.551808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.551898 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.551970 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.552886 (+ 916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33124: BlockingRecv error: recv got EOF from 127.0.0.1:33124 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 06:39:24.561488 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.559569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.28.104.253:36199)
0619 06:39:24.559733 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.559741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.559880 (+ 139us) server_negotiation.cc:407] Connection header received
0619 06:39:24.560188 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.560202 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.560319 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.560422 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.561256 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33134: BlockingRecv error: recv got EOF from 127.0.0.1:33134 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 06:39:24.576239 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.566688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.28.104.253:36199)
0619 06:39:24.566857 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.566863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.574741 (+ 7878us) server_negotiation.cc:407] Connection header received
0619 06:39:24.574945 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.574951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.575044 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.575148 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.576012 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33142: BlockingRecv error: recv got EOF from 127.0.0.1:33142 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:39:24.583308 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.581552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.28.104.253:36199)
0619 06:39:24.581700 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.581706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.581800 (+ 94us) server_negotiation.cc:407] Connection header received
0619 06:39:24.582059 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.582065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.582158 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.582269 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.583056 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33154: BlockingRecv error: recv got EOF from 127.0.0.1:33154 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 06:39:24.595043 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.590771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.28.104.253:36199)
0619 06:39:24.590965 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.590971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.593596 (+ 2625us) server_negotiation.cc:407] Connection header received
0619 06:39:24.593854 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.593861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.593963 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.594082 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.594828 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33170: BlockingRecv error: recv got EOF from 127.0.0.1:33170 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 06:39:24.612632 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.610510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.28.104.253:36199)
0619 06:39:24.610677 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.610684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.611188 (+ 504us) server_negotiation.cc:407] Connection header received
0619 06:39:24.611376 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.611385 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.611478 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.611595 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.612405 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33172: BlockingRecv error: recv got EOF from 127.0.0.1:33172 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 06:39:24.634895 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.632164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.28.104.253:36199)
0619 06:39:24.632378 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.632385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.632933 (+ 548us) server_negotiation.cc:407] Connection header received
0619 06:39:24.633632 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.633640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.633741 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.633853 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.634657 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33174: BlockingRecv error: recv got EOF from 127.0.0.1:33174 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 06:39:24.673677 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.671379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.28.104.253:36199)
0619 06:39:24.671551 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.671557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.672089 (+ 532us) server_negotiation.cc:407] Connection header received
0619 06:39:24.672291 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.672297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.672426 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.672548 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.673420 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33180: BlockingRecv error: recv got EOF from 127.0.0.1:33180 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 06:39:24.742903 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.740994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.28.104.253:36199)
0619 06:39:24.741246 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.741254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.741581 (+ 327us) server_negotiation.cc:407] Connection header received
0619 06:39:24.741748 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.741754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.741849 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.741972 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.742686 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33196: BlockingRecv error: recv got EOF from 127.0.0.1:33196 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260619 06:39:24.876093 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:24.874333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.28.104.253:36199)
0619 06:39:24.874507 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:24.874512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:24.874901 (+ 389us) server_negotiation.cc:407] Connection header received
0619 06:39:24.875056 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:24.875060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:24.875131 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:24.875228 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:24.875894 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33204: BlockingRecv error: recv got EOF from 127.0.0.1:33204 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:39:25.104111 19384 catalog_manager.cc:1259] Loaded cluster ID: 805b092af41e4c3c8327f98bcd6e32bb
I20260619 06:39:25.104297 19384 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549: loading cluster ID for follower catalog manager: success
I20260619 06:39:25.107082 19384 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549: acquiring CA information for follower catalog manager: success
I20260619 06:39:25.108994 19384 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd92679848d84134b1d7338c6f392549: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:39:25.139725 19296 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:25.137818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.28.104.253:36199)
0619 06:39:25.138058 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:25.138064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:25.138277 (+ 213us) server_negotiation.cc:407] Connection header received
0619 06:39:25.138523 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:25.138530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:25.138621 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:25.138758 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:25.139493 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33214: BlockingRecv error: recv got EOF from 127.0.0.1:33214 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:39:25.502591 19437 catalog_manager.cc:1259] Loaded cluster ID: 805b092af41e4c3c8327f98bcd6e32bb
I20260619 06:39:25.502749 19437 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418: loading cluster ID for follower catalog manager: success
I20260619 06:39:25.506399 19437 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418: acquiring CA information for follower catalog manager: success
I20260619 06:39:25.508651 19437 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ba420c7beb24f33adf0568c6f84b418: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:39:25.662694 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:25.655428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.28.104.253:36199)
0619 06:39:25.656206 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:25.656212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:25.656234 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:39:25.656311 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:25.656322 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:25.656519 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:25.656663 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:25.657065 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:25.658336 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:25.659866 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:25.660609 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:25.660739 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:25.661234 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:25.661973 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:25.662139 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:25.662262 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 06:39:25.662354 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":449,"threads_started":1}
I20260619 06:39:25.680894 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:25.675025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.28.104.252:34833)
0619 06:39:25.675698 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:25.675704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:25.675738 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:39:25.675850 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:25.675854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:25.675995 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:25.676172 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:25.676599 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:25.677362 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:25.678651 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:25.679057 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:25.679195 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:25.679679 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:25.680233 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:25.680398 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:25.680454 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 06:39:25.680537 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":433,"threads_started":1}
I20260619 06:39:25.708614 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:25.688828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.28.104.252:34833)
0619 06:39:25.688965 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:25.688973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:25.701528 (+ 12555us) server_negotiation.cc:407] Connection header received
0619 06:39:25.702033 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:25.702041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:25.702152 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:25.702230 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:25.703080 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:25.704177 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:25.706433 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:25.707077 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:25.707173 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:25.707760 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:25.707900 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:25.707961 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:25.708208 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 06:39:25.708292 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 06:39:25.709844 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:25.688174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.28.104.253:36199)
0619 06:39:25.688933 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:25.688939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:25.702934 (+ 13995us) server_negotiation.cc:407] Connection header received
0619 06:39:25.703101 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:25.703106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:25.703207 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:25.703295 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:25.703970 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:25.705239 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:25.707420 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:25.708054 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:25.708180 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:25.708783 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:25.708916 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:25.708979 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:25.709499 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 06:39:25.709578 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:39:25.710029 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:25.688515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.28.104.254:38515)
0619 06:39:25.701673 (+ 13158us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:25.701680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:25.701699 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:39:25.701806 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:25.701812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:25.702015 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:25.702203 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:25.703628 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:25.704693 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:25.707036 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:25.707685 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:25.707810 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:25.708508 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:25.708934 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:25.709061 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:25.709646 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 06:39:25.709731 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13026,"thread_start_us":12914,"threads_started":1}
I20260619 06:39:25.715518 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 19150
I20260619 06:39:25.716627 19212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:25.829156 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 19150
I20260619 06:39:25.850862 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 19224
I20260619 06:39:25.852064 19287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:25.963143 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 19224
I20260619 06:39:25.980847 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 19299
I20260619 06:39:25.982017 19361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:26.089280 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 19299
2026-06-19T06:39:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4899 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:26 dist-test-slave-nf4n krb5kdc[19521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:26 dist-test-slave-nf4n krb5kdc[19521](info): set up 2 sockets
Jun 19 06:39:26 dist-test-slave-nf4n krb5kdc[19521](info): commencing operation
krb5kdc: starting...
W20260619 06:39:27.217250 29091 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/test-user.keytab.
Jun 19 06:39:27 dist-test-slave-nf4n krb5kdc[19521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.104.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
Entry for principal kudu/127.28.104.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab.
I20260619 06:39:27.482674 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:41367
--webserver_interface=127.28.104.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:41367,127.28.104.253:37279,127.28.104.252:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:27.810421 19535 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:39:27.810734 19535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:27.810825 19535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:27.810966 19535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:27.818075 19535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:27.818154 19535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:27.818210 19535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:27.818252 19535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:27.818369 19535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:27.827837 19535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:41367,127.28.104.253:37279,127.28.104.252:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.254.keytab
--principal=kudu/127.28.104.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.254:41367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.104.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:27.829097 19535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:27.830376 19535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:27.842235 19541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:27.842618 19540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:27.842697 19543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:27.843300 19535 server_base.cc:1034] running on GCE node
I20260619 06:39:27.844110 19535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:27.845685 19535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:27.846907 19535 hybrid_clock.cc:648] HybridClock initialized: now 1781851167846837 us; error 45 us; skew 500 ppm
I20260619 06:39:27.847237 19535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:27 dist-test-slave-nf4n krb5kdc[19521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851167, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:27.850751 19535 init.cc:377] Logged in from keytab as kudu/127.28.104.254@KRBTEST.COM (short username kudu)
I20260619 06:39:27.852134 19535 webserver.cc:460] Webserver started at http://127.28.104.254:38669/ using document root <none> and password file <none>
I20260619 06:39:27.852846 19535 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:27.852957 19535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:27.853318 19535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:27.856328 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/data/instance:
uuid: "234d24401a8340679bb59f6a95cd657e"
format_stamp: "Formatted at 2026-06-19 06:39:27 on dist-test-slave-nf4n"
server_key: "362922bc7b0d7cc3198c4e0ede3e4def"
server_key_iv: "c190efbe367e23e0300d6ba29d23644b"
server_key_version: "encryptionkey@0"
I20260619 06:39:27.857194 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/wal/instance:
uuid: "234d24401a8340679bb59f6a95cd657e"
format_stamp: "Formatted at 2026-06-19 06:39:27 on dist-test-slave-nf4n"
server_key: "362922bc7b0d7cc3198c4e0ede3e4def"
server_key_iv: "c190efbe367e23e0300d6ba29d23644b"
server_key_version: "encryptionkey@0"
I20260619 06:39:27.862205 19535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:39:27.865571 19550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:27.866940 19535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:39:27.867123 19535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/wal
uuid: "234d24401a8340679bb59f6a95cd657e"
format_stamp: "Formatted at 2026-06-19 06:39:27 on dist-test-slave-nf4n"
server_key: "362922bc7b0d7cc3198c4e0ede3e4def"
server_key_iv: "c190efbe367e23e0300d6ba29d23644b"
server_key_version: "encryptionkey@0"
I20260619 06:39:27.867336 19535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:27.905190 19535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:27.909480 19535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:27.909781 19535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:27.928232 19535 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.254:41367
I20260619 06:39:27.928320 19601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.254:41367 every 8 connection(s)
I20260619 06:39:27.929636 19535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:27.933475 19602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:27.937548 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 19535
I20260619 06:39:27.937747 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-0/wal/instance
I20260619 06:39:27.938159 29091 external_mini_cluster.cc:1428] Setting key 1c030896512756e933a66424f41467c5
I20260619 06:39:27.939998 19602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:39:27 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851167, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.104.253@KRBTEST.COM; defaulting to no policy
I20260619 06:39:27.962435 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:27.943180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:41367 (local address 127.0.0.1:43118)
0619 06:39:27.944314 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:27.944357 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:27.946206 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:27.947029 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:27.947035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:27.947748 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:27.948077 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:27.948093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:27.949711 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:27.949715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:27.950769 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:27.950780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:27.951702 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:27.951708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:27.951882 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:27.952566 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:27.952587 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:27.954982 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:27.958326 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:27.958339 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:27.958350 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:27.958739 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:27.959128 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:27.959134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:27.959137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:27.959289 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:27.960012 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:27.960020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:27.960219 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 06:39:27.960464 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 06:39:27.961060 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":146,"spinlock_wait_cycles":17792,"thread_start_us":725,"threads_started":1}
I20260619 06:39:27.962467 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:27.943174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.28.104.254:41367)
0619 06:39:27.944003 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:27.944022 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:27.944476 (+ 454us) server_negotiation.cc:407] Connection header received
0619 06:39:27.946568 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:27.946573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:27.946810 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:27.947019 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:27.948284 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:27.949547 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:27.950908 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:27.951454 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:27.951662 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:27.955206 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:27.955214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:27.955226 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:27.955256 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:27.958193 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:27.958867 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:27.958876 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:27.958879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:27.959022 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:27.959392 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:27.959400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:27.959403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:27.959872 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:27.960073 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:27.960571 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 06:39:27.960926 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":591,"thread_start_us":389,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.253@KRBTEST.COM" created.
I20260619 06:39:27.965443 19602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:39:27.966428 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:27.966082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50558)
0619 06:39:27.966230 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:27.966326 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 06:39:27.966807 19553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
I20260619 06:39:27.968370 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:27.968034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50574)
0619 06:39:27.968179 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:27.968251 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:39:27.968549 19602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:37279: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab.
Entry for principal kudu/127.28.104.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab.
I20260619 06:39:27.991555 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.253:37279
--webserver_interface=127.28.104.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab
--principal=kudu/127.28.104.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:41367,127.28.104.253:37279,127.28.104.252:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:39:28.017921 19602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } attempt: 1
I20260619 06:39:28.019093 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.018624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50582)
0619 06:39:28.018852 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.018964 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 06:39:28.020197 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.019853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50592)
0619 06:39:28.020009 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.020072 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 06:39:28.020416 19602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:37279: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
I20260619 06:39:28.088699 19602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } attempt: 2
I20260619 06:39:28.090315 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.089498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50606)
0619 06:39:28.089882 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.089989 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260619 06:39:28.091513 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.091138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50620)
0619 06:39:28.091308 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.091381 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:39:28.091713 19602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:37279: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
I20260619 06:39:28.182978 19602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } attempt: 3
I20260619 06:39:28.184082 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.183641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50624)
0619 06:39:28.183845 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.183944 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 06:39:28.185256 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.184857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50634)
0619 06:39:28.185011 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.185127 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:39:28.185439 19602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:37279: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
W20260619 06:39:28.321323 19609 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:39:28.321653 19609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:28.321789 19609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:28.321909 19609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 06:39:28.328693 19602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } attempt: 4
W20260619 06:39:28.329391 19609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:28.329504 19609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:28.329638 19609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:28.329737 19609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:28.329830 19609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:28.330015 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.329402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50636)
0619 06:39:28.329737 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.329844 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260619 06:39:28.331000 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.330689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50650)
0619 06:39:28.330816 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.330895 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:39:28.331202 19602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.253:37279: Network error: Client connection negotiation failed: client connection to 127.28.104.253:37279: connect: Connection refused (error 111)
I20260619 06:39:28.340447 19609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:41367,127.28.104.253:37279,127.28.104.252:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.253.keytab
--principal=kudu/127.28.104.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.253:37279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.104.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:28.341841 19609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:28.343186 19609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:28.354655 19614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:28.355957 19609 server_base.cc:1034] running on GCE node
W20260619 06:39:28.355937 19617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:28.355963 19615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:28.357000 19609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:28.358409 19609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:28.359647 19609 hybrid_clock.cc:648] HybridClock initialized: now 1781851168359583 us; error 48 us; skew 500 ppm
I20260619 06:39:28.359973 19609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:28 dist-test-slave-nf4n krb5kdc[19521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:28.362949 19609 init.cc:377] Logged in from keytab as kudu/127.28.104.253@KRBTEST.COM (short username kudu)
I20260619 06:39:28.364295 19609 webserver.cc:460] Webserver started at http://127.28.104.253:45457/ using document root <none> and password file <none>
I20260619 06:39:28.364970 19609 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:28.365113 19609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:28.365448 19609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:28.368377 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/data/instance:
uuid: "37eee7af3728462699f4e205f2830245"
format_stamp: "Formatted at 2026-06-19 06:39:28 on dist-test-slave-nf4n"
server_key: "843f3334275cfcebbea3fbfac0117077"
server_key_iv: "01a2ff8e7b70ecc5049a9e4e891f2dfd"
server_key_version: "encryptionkey@0"
I20260619 06:39:28.369233 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/wal/instance:
uuid: "37eee7af3728462699f4e205f2830245"
format_stamp: "Formatted at 2026-06-19 06:39:28 on dist-test-slave-nf4n"
server_key: "843f3334275cfcebbea3fbfac0117077"
server_key_iv: "01a2ff8e7b70ecc5049a9e4e891f2dfd"
server_key_version: "encryptionkey@0"
I20260619 06:39:28.374302 19609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 06:39:28.377722 19624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:28.379077 19609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:28.379287 19609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/wal
uuid: "37eee7af3728462699f4e205f2830245"
format_stamp: "Formatted at 2026-06-19 06:39:28 on dist-test-slave-nf4n"
server_key: "843f3334275cfcebbea3fbfac0117077"
server_key_iv: "01a2ff8e7b70ecc5049a9e4e891f2dfd"
server_key_version: "encryptionkey@0"
I20260619 06:39:28.379513 19609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:28.410421 19609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:28.414283 19609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:28.414558 19609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:28.432255 19609 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.253:37279
I20260619 06:39:28.432277 19675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.253:37279 every 8 connection(s)
I20260619 06:39:28.433593 19609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/data/info.pb
I20260619 06:39:28.437197 19676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:28.438371 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 19609
I20260619 06:39:28.438556 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-1/wal/instance
I20260619 06:39:28.438980 29091 external_mini_cluster.cc:1428] Setting key ae15191e0d76d6c19489d1d0ea3b5a5d
I20260619 06:39:28.443589 19676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:39:28 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.104.252@KRBTEST.COM; defaulting to no policy
I20260619 06:39:28.463582 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.446118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.28.104.254:41367)
0619 06:39:28.446335 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:28.446343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:28.447794 (+ 1451us) server_negotiation.cc:407] Connection header received
0619 06:39:28.449684 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:28.449698 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:28.449788 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:28.449910 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:28.451322 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.452008 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.453621 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.454046 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.454183 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.457805 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:28.457810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:28.457813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:28.457846 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:28.460549 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.461337 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.461343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.461345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.461464 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.461869 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.461875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.461880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.462200 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:28.462363 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:28.463036 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 06:39:28.463256 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.104.252@KRBTEST.COM" created.
I20260619 06:39:28.464654 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.446866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:41367 (local address 127.0.0.1:43128)
0619 06:39:28.447630 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.447672 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:28.449411 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:28.450128 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:28.450136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:28.450635 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:28.451141 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.451157 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.452171 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.452175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:28.453466 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.453478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.454283 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.454290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:28.454516 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.455211 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:28.455233 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:28.457575 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:28.460692 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:28.460700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:28.460717 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:28.461210 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:28.461573 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:28.461579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:28.461583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:28.461752 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:28.462330 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:28.462340 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:28.462678 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 06:39:28.462934 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 06:39:28.463441 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":324,"threads_started":1}
I20260619 06:39:28.466394 19676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:39:28 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
I20260619 06:39:28.482054 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.467033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50660)
0619 06:39:28.467235 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.467250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:28.467330 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:28.468454 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:28.468459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:28.468882 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:28.469004 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.469011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.471140 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.471145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:28.472622 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.472637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.473570 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.473576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:28.473770 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.474347 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:28.474362 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:28.476448 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:28.479652 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:28.479657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:28.479661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:28.480001 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:28.480368 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:28.480374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:28.480380 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:28.480521 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:28.481097 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:28.481101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:28.481234 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 06:39:28.481394 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 06:39:28.481663 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 06:39:28.482163 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.467395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.28.104.253:37279)
0619 06:39:28.468060 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:28.468066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:28.468091 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:39:28.468133 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:28.468138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:28.468311 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:28.468469 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:28.469220 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.470996 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.472795 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.473432 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.473566 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.476650 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:28.476670 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:28.476682 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:28.476708 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:28.479505 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.480126 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.480133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.480137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.480254 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.480620 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.480627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.480631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.480949 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:28.481220 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:28.481538 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 06:39:28.481765 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":533,"thread_start_us":387,"threads_started":1}
I20260619 06:39:28.484961 19676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:39:28.486132 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.485694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44416)
0619 06:39:28.485855 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.485984 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:39:28.486549 19625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
I20260619 06:39:28.488021 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.487724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44426)
0619 06:39:28.487853 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.487926 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:39:28.488319 19676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:43591: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.104.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab.
Entry for principal kudu/127.28.104.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab.
I20260619 06:39:28.496515 29091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
/tmp/dist-test-taskS2JAv5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.252:43591
--webserver_interface=127.28.104.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab
--principal=kudu/127.28.104.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.104.212:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.104.254:41367,127.28.104.253:37279,127.28.104.252:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:39:28.537609 19676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } attempt: 1
I20260619 06:39:28.538838 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.538392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44438)
0619 06:39:28.538587 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.538693 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 06:39:28.539800 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.539454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44442)
0619 06:39:28.539621 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.539695 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 06:39:28.540012 19676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:43591: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
I20260619 06:39:28.608225 19676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } attempt: 2
I20260619 06:39:28.609455 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.608953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44450)
0619 06:39:28.609213 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.609303 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 06:39:28.610426 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.610088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44452)
0619 06:39:28.610247 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.610315 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:39:28.610623 19676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:43591: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
I20260619 06:39:28.630442 19602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } attempt: 5
Jun 19 06:39:28 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851167, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
I20260619 06:39:28.645134 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.631303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.28.104.253:37279)
0619 06:39:28.631506 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:28.631512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:28.631676 (+ 164us) server_negotiation.cc:407] Connection header received
0619 06:39:28.631922 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:28.631926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:28.632010 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:28.632149 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:28.633103 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.633878 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.635162 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.635678 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.635821 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.639416 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:28.639421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:28.639425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:28.639460 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:28.642365 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.643061 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.643078 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.643081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.643205 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.643643 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.643649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.643657 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.643940 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:28.644116 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:28.644512 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 06:39:28.644744 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 06:39:28.645138 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.631261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50674)
0619 06:39:28.631507 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.631531 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:28.631763 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:28.632181 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:28.632186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:28.632691 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:28.632902 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.632912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.634019 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.634027 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:28.634993 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.635006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.635838 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.635844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:28.636087 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.636771 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:28.636792 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:28.639207 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:28.642506 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:28.642514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:28.642518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:28.642926 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:28.643322 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:28.643328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:28.643332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:28.643509 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:28.644063 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:28.644068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:28.644244 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 06:39:28.644400 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 06:39:28.644723 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 06:39:28.646734 19602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:39:28.647864 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.647451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44462)
0619 06:39:28.647638 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.647731 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 06:39:28.648978 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.648630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44464)
0619 06:39:28.648805 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.648874 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 06:39:28.649281 19602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:43591: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
I20260619 06:39:28.700510 19602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } attempt: 1
I20260619 06:39:28.701781 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.701277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44480)
0619 06:39:28.701475 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.701579 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 06:39:28.701839 19676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } attempt: 3
I20260619 06:39:28.702814 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.702403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44496)
0619 06:39:28.702592 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.702682 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 06:39:28.703585 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.703165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44502)
0619 06:39:28.703340 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.703430 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 06:39:28.703809 19602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:43591: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
I20260619 06:39:28.703951 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.703505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44514)
0619 06:39:28.703710 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.703806 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260619 06:39:28.704222 19676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:43591: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
I20260619 06:39:28.772055 19602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } attempt: 2
I20260619 06:39:28.773330 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.772781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44516)
0619 06:39:28.772983 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.773163 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 06:39:28.774498 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.774056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44532)
0619 06:39:28.774246 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.774346 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:39:28.774857 19602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:43591: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
W20260619 06:39:28.837733 19684 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:39:28.838153 19684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:28.838294 19684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:28.838454 19684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:28.846525 19684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:28.846668 19684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:28.846773 19684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:28.846877 19684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:28.846975 19684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:28.847429 19676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } attempt: 4
I20260619 06:39:28.848624 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.848104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44538)
0619 06:39:28.848359 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.848468 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 06:39:28.849630 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.849300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44550)
0619 06:39:28.849450 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.849528 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:39:28.849835 19676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:43591: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
I20260619 06:39:28.857905 19684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.104.212:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.104.254:41367,127.28.104.253:37279,127.28.104.252:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/krb5kdc/kudu_127.28.104.252.keytab
--principal=kudu/127.28.104.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.104.252:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.104.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:28.859282 19684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:28.860751 19684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:28.873237 19689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:28.873171 19690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:28.873263 19692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:28.874018 19684 server_base.cc:1034] running on GCE node
I20260619 06:39:28.874766 19684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:28.876106 19684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:28.877326 19684 hybrid_clock.cc:648] HybridClock initialized: now 1781851168877262 us; error 31 us; skew 500 ppm
I20260619 06:39:28.877656 19684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:28 dist-test-slave-nf4n krb5kdc[19521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:28.880705 19684 init.cc:377] Logged in from keytab as kudu/127.28.104.252@KRBTEST.COM (short username kudu)
I20260619 06:39:28.881096 19602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } attempt: 3
I20260619 06:39:28.882308 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.881773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44552)
0619 06:39:28.881988 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.882069 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:39:28.882380 19684 webserver.cc:460] Webserver started at http://127.28.104.252:40705/ using document root <none> and password file <none>
I20260619 06:39:28.883288 19684 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:28.883376 19684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:28.883384 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.883003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44558)
0619 06:39:28.883184 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.883253 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260619 06:39:28.883630 19602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.104.252:43591: Network error: Client connection negotiation failed: client connection to 127.28.104.252:43591: connect: Connection refused (error 111)
I20260619 06:39:28.883729 19684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:28.886721 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/data/instance:
uuid: "e745c2835f224f55b694b504d66a3d42"
format_stamp: "Formatted at 2026-06-19 06:39:28 on dist-test-slave-nf4n"
server_key: "fe2baae6788cb44314da4c1b8adb4524"
server_key_iv: "2158242874be5ab02b7a3bb036ea1846"
server_key_version: "encryptionkey@0"
I20260619 06:39:28.887490 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/wal/instance:
uuid: "e745c2835f224f55b694b504d66a3d42"
format_stamp: "Formatted at 2026-06-19 06:39:28 on dist-test-slave-nf4n"
server_key: "fe2baae6788cb44314da4c1b8adb4524"
server_key_iv: "2158242874be5ab02b7a3bb036ea1846"
server_key_version: "encryptionkey@0"
I20260619 06:39:28.892359 19684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:39:28.895710 19699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:28.896961 19684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:39:28.897166 19684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/wal
uuid: "e745c2835f224f55b694b504d66a3d42"
format_stamp: "Formatted at 2026-06-19 06:39:28 on dist-test-slave-nf4n"
server_key: "fe2baae6788cb44314da4c1b8adb4524"
server_key_iv: "2158242874be5ab02b7a3bb036ea1846"
server_key_version: "encryptionkey@0"
I20260619 06:39:28.897384 19684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:28.923434 19684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:28.927359 19684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:28.927618 19684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:28.946100 19684 rpc_server.cc:307] RPC server started. Bound to: 127.28.104.252:43591
I20260619 06:39:28.946122 19750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.104.252:43591 every 8 connection(s)
I20260619 06:39:28.947531 19684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/data/info.pb
I20260619 06:39:28.951290 19751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:28.951542 29091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu as pid 19684
I20260619 06:39:28.951732 29091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2JAv5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850805520649-29091-0/minicluster-data/master-2/wal/instance
I20260619 06:39:28.952158 29091 external_mini_cluster.cc:1428] Setting key d40180cc52a69e693ef06631a0f16f0e
I20260619 06:39:28.961196 19751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:39:28 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
Jun 19 06:39:28 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
I20260619 06:39:28.975494 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.953906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.28.104.252:43591)
0619 06:39:28.954751 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:28.954767 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:28.954822 (+ 55us) server_negotiation.cc:407] Connection header received
0619 06:39:28.956394 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:28.956427 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:28.956950 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:28.957421 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:28.958608 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.960546 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.961821 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.962775 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.962945 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.967550 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:28.967557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:28.967567 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:28.967604 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:28.971217 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.971886 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.971894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.971913 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.972065 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.972463 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.972468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.972473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.973214 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:28.973569 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:28.974056 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 06:39:28.974373 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":353,"threads_started":1}
I20260619 06:39:28.979045 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.963881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.254:41367 (local address 127.0.0.1:43130)
0619 06:39:28.964880 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.964919 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:28.965146 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:28.965539 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:28.965547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:28.966313 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:28.966623 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.966636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.967658 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.967673 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:28.969121 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.969137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.970162 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.970167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:28.970283 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.970944 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:28.970960 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:28.973414 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:28.976612 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:28.976624 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:28.976631 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:28.977069 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:28.977428 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:28.977433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:28.977437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:28.977565 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:28.978098 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:28.978106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:28.978300 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 06:39:28.978494 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 06:39:28.978780 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":668,"threads_started":1}
I20260619 06:39:28.979912 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.963969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.28.104.254:41367)
0619 06:39:28.964608 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:28.964617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:28.965121 (+ 504us) server_negotiation.cc:407] Connection header received
0619 06:39:28.965292 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:28.965304 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:28.965367 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:28.965481 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:28.966789 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.967490 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.969299 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.969970 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.970100 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.973658 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:28.973665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:28.973670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:28.973710 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:28.976485 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.977177 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.977182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.977185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.977317 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.977696 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.977702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.977710 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.977959 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:28.978115 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:28.979492 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0619 06:39:28.979645 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":367,"threads_started":1}
I20260619 06:39:28.981302 19751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:39:28 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.254@KRBTEST.COM
Jun 19 06:39:28 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
I20260619 06:39:28.993816 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.978259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43144 (local address 127.28.104.254:41367)
0619 06:39:28.978839 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:28.978843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:28.978857 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:39:28.978892 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:28.978895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:28.978947 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:28.979056 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:28.980251 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.981283 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.982673 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.983131 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.983308 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.987345 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:28.987351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:28.987358 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:28.987398 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:28.991072 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.991698 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.991705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.991709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.991857 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.992186 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.992194 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.992198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.992571 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:28.992766 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:28.993174 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 06:39:28.993394 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":267,"threads_started":1}
I20260619 06:39:28.995121 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.981899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.253:37279 (local address 127.0.0.1:50690)
0619 06:39:28.982029 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.982046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:28.982225 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:28.982586 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:28.982590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:28.982946 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:28.983072 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.983079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.984345 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.984351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:28.985285 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.985296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.986013 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.986018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:28.986165 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.986632 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:28.986648 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:28.988813 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:28.992505 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:28.992508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:28.992511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:28.992993 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:28.993452 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:28.993457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:28.993460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:28.993636 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:28.994210 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:28.994215 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:28.994374 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 06:39:28.994512 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 06:39:28.994783 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 06:39:28.995170 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.982041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.28.104.253:37279)
0619 06:39:28.982325 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:28.982331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:28.982348 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:39:28.982405 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:28.982409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:28.982465 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:28.982566 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:28.983291 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.984162 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.985442 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.985873 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.985998 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:28.989181 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:28.989191 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:28.989199 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:28.989242 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:28.992382 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.993165 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.993171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.993174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.993328 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:28.993766 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:28.993771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:28.993774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:28.994095 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:28.994257 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:28.994600 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 06:39:28.994810 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260619 06:39:28.996276 19751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:39:28 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.252@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
Jun 19 06:39:29 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.104.253@KRBTEST.COM
I20260619 06:39:29.011206 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.997113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44574)
0619 06:39:28.997268 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:28.997286 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:28.997374 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:28.997823 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:28.997828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:28.998248 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:28.998412 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:28.998422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.999740 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:28.999744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:29.001049 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:29.001064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.001997 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.002004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:29.002251 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.002866 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:29.002893 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:29.005331 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:29.008577 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:29.008583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:29.008588 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:29.009117 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:29.009478 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:29.009484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:29.009488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:29.009645 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:29.010328 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:29.010332 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:29.010446 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 06:39:29.010556 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 06:39:29.010845 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 06:39:29.011499 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.997386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.28.104.252:43591)
0619 06:39:28.997577 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:28.997582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:28.997598 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:39:28.997650 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:28.997653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:28.997702 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:28.997770 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:28.998537 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.999602 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.001196 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.001826 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.001921 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.005577 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:29.005584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:29.005590 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:29.005632 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:29.008443 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.009227 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.009231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.009233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.009358 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.009804 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.009807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.009809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.010203 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:29.010366 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.010727 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.011012 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":60}
I20260619 06:39:29.013207 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:28.995572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.28.104.253:37279)
0619 06:39:28.995728 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:28.995733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:28.995854 (+ 121us) server_negotiation.cc:407] Connection header received
0619 06:39:28.996039 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:28.996043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:28.996111 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:28.996209 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:28.997575 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:28.998583 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.000081 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.000750 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.000873 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.006721 (+ 5848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:29.006726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:29.006729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:29.006759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:29.009923 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.010740 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.010747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.010753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.010908 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.011393 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.011398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.011402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.011821 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:29.012053 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.012567 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.012814 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:39:29.016836 19751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42: Bootstrap starting.
I20260619 06:39:29.018313 29091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:29.020515 19751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:29.021920 19751 log.cc:824] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:29.020704 19577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "234d24401a8340679bb59f6a95cd657e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43144
I20260619 06:39:29.025202 19751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42: No bootstrap required, opened a new log
I20260619 06:39:29.033610 19751 raft_consensus.cc:348] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } }
I20260619 06:39:29.033932 19751 raft_consensus.cc:729] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e745c2835f224f55b694b504d66a3d42, State: Initialized, Role: FOLLOWER
I20260619 06:39:29.034572 19751 consensus_queue.cc:260] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } }
I20260619 06:39:29.035501 19759 sys_catalog.cc:455] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } } }
I20260619 06:39:29.035835 19759 sys_catalog.cc:458] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:29.036926 19751 sys_catalog.cc:564] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:39:29.047399 19770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:39:29.047657 19770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:39:29.045176 19684 master_runner.cc:186] Error getting master registration for 127.28.104.254:41367: OK, instance_id { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" instance_seqno: 1781851167922893 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:39:29.054117 19684 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:39:29.060868 19602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } attempt: 4
Jun 19 06:39:29 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851167, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.254@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
I20260619 06:39:29.077736 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.061742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.28.104.252:43591)
0619 06:39:29.061977 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.061983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.062089 (+ 106us) server_negotiation.cc:407] Connection header received
0619 06:39:29.062279 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.062285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.062366 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.062473 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.063555 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.064762 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.066733 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.067304 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.067450 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.070962 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:29.070967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:29.070971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:29.071004 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:29.074855 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.075562 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.075569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.075573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.075710 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.076132 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.076140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.076144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.076467 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:29.076665 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.077106 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.077314 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:39:29.077732 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.061755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44590)
0619 06:39:29.062007 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:29.062026 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:29.062146 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:29.062529 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:29.062537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:29.063114 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:29.063379 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:29.063393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.064923 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.064931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:29.066588 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:29.066609 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.067480 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.067487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:29.067716 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.068361 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:29.068380 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:29.070740 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:29.075012 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:29.075019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:29.075023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:29.075421 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:29.075843 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:29.075848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:29.075852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:29.076007 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:29.076606 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:29.076612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:29.076797 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 06:39:29.076966 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 06:39:29.077297 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 06:39:29.082677 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e: Bootstrap starting.
I20260619 06:39:29.087090 19602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:29.088428 19602 log.cc:824] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:29.091451 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e: No bootstrap required, opened a new log
I20260619 06:39:29.099249 19602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } }
I20260619 06:39:29.099669 19602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 234d24401a8340679bb59f6a95cd657e, State: Initialized, Role: FOLLOWER
I20260619 06:39:29.100118 19602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } }
I20260619 06:39:29.101140 19773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } } }
I20260619 06:39:29.101656 19773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:29.102357 19602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:29.102463 19577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 06:39:29.102690 19577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:29.106053 19577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } }
I20260619 06:39:29.108428 19726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "234d24401a8340679bb59f6a95cd657e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e745c2835f224f55b694b504d66a3d42"
I20260619 06:39:29.108907 19726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:29.108856 19577 leader_election.cc:288] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [CANDIDATE]: Term 1 election: Requested vote from peers 37eee7af3728462699f4e205f2830245 (127.28.104.253:37279), e745c2835f224f55b694b504d66a3d42 (127.28.104.252:43591)
I20260619 06:39:29.112644 19726 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 234d24401a8340679bb59f6a95cd657e in term 1.
I20260619 06:39:29.113466 19551 leader_election.cc:302] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 234d24401a8340679bb59f6a95cd657e, e745c2835f224f55b694b504d66a3d42; no voters:
I20260619 06:39:29.114025 19773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:29.125097 19773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [term 1 LEADER]: Becoming Leader. State: Replica: 234d24401a8340679bb59f6a95cd657e, State: Running, Role: LEADER
I20260619 06:39:29.125861 19773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } }
I20260619 06:39:29.134343 19778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 234d24401a8340679bb59f6a95cd657e. Latest consensus state: current_term: 1 leader_uuid: "234d24401a8340679bb59f6a95cd657e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } } }
I20260619 06:39:29.134907 19778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:29.139294 19784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:29.149034 19676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } attempt: 5
W20260619 06:39:29.162735 19556 tablet.cc:2369] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:39:29 dist-test-slave-nf4n krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851168, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.104.253@KRBTEST.COM for kudu/127.28.104.252@KRBTEST.COM
I20260619 06:39:29.173821 19784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 06:39:29.179849 19705 tablet.cc:2369] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:39:29.181586 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.149641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.104.252:43591 (local address 127.0.0.1:44612)
0619 06:39:29.149921 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:29.149940 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:29.150062 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:29.155422 (+ 5360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:29.155429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:29.155982 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:29.156219 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:29.156233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.160016 (+ 3783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.160025 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:29.161569 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:29.161587 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.163481 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.163487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:29.163749 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.164443 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:29.164470 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:29.167936 (+ 3466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:29.177588 (+ 9652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:29.177594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:29.177598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:29.178120 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:29.179021 (+ 901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:29.179029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:29.179032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:29.179245 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:29.180505 (+ 1260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:29.180510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:29.180690 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 06:39:29.180836 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 06:39:29.181166 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260619 06:39:29.184744 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.150703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.28.104.252:43591)
0619 06:39:29.154351 (+ 3648us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.154357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.154381 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:39:29.154429 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.154438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.154502 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.156402 (+ 1900us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.156708 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.158072 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.161819 (+ 3747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.162479 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.166090 (+ 3611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.168181 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:29.168192 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:29.168197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:29.168237 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:29.173159 (+ 4922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.178457 (+ 5298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.178463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.178468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.178670 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.179415 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.179420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.179426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.179800 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:29.182925 (+ 3125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.183360 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.183648 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3516,"thread_start_us":3343,"threads_started":1}
I20260619 06:39:29.191610 19676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245: Bootstrap starting.
I20260619 06:39:29.192610 19726 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [term 1 FOLLOWER]: Refusing update from remote peer 234d24401a8340679bb59f6a95cd657e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:29.193742 19778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:29.201205 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.145761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.28.104.254:41367)
0619 06:39:29.145987 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.145996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.155718 (+ 9722us) server_negotiation.cc:407] Connection header received
0619 06:39:29.156950 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.156957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.157058 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.157135 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.162081 (+ 4946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.163366 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.171274 (+ 7908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.172696 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.172817 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.188117 (+ 15300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:29.188123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:29.188127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:29.188162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:29.195321 (+ 7159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.196868 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.196874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.196878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.197058 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.198163 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.198179 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.198185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.198542 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:29.198784 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.200491 (+ 1707us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.200747 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:39:29.204087 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.154319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.28.104.253:37279)
0619 06:39:29.154480 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.154486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.156395 (+ 1909us) server_negotiation.cc:407] Connection header received
0619 06:39:29.159523 (+ 3128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.159531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.159629 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.159723 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.165131 (+ 5408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.166278 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.171351 (+ 5073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.171820 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.171890 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.188108 (+ 16218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:29.188115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:29.188119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:29.188163 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:29.193338 (+ 5175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.197548 (+ 4210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.197555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.197558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.197732 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.199280 (+ 1548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.199285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.199289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.199681 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:29.199923 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.203433 (+ 3510us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.203690 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:39:29.204905 19676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:29.207597 19676 log.cc:824] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:29.219405 19759 sys_catalog.cc:455] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 234d24401a8340679bb59f6a95cd657e. Latest consensus state: current_term: 1 leader_uuid: "234d24401a8340679bb59f6a95cd657e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } } }
I20260619 06:39:29.219761 19759 sys_catalog.cc:458] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:29.223206 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.140795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.28.104.252:43591)
0619 06:39:29.140990 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.140997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.153533 (+ 12536us) server_negotiation.cc:407] Connection header received
0619 06:39:29.153760 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.153766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.153853 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.153960 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.159024 (+ 5064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.160136 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.165398 (+ 5262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.166179 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.166276 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.182148 (+ 15872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:29.182155 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:29.182160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:29.182199 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:29.185882 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.210908 (+ 25026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.210948 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.210953 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.211288 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:29.219930 (+ 8642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:29.219937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:29.219941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:29.220313 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:29.220667 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.222547 (+ 1880us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.222837 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:39:29.225381 19800 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:39:29.226722 19676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245: No bootstrap required, opened a new log
I20260619 06:39:29.229130 19795 mvcc.cc:205] Tried to move back new op lower bound from 7298462388976644096 to 7298462388747075584. Current Snapshot: MvccSnapshot[applied={T|T < 7298462388976644096}]
I20260619 06:39:29.239164 19759 sys_catalog.cc:455] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "234d24401a8340679bb59f6a95cd657e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } } }
I20260619 06:39:29.239432 19759 sys_catalog.cc:458] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:29.246069 19676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } }
I20260619 06:39:29.246539 19676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37eee7af3728462699f4e205f2830245, State: Initialized, Role: FOLLOWER
I20260619 06:39:29.247128 19676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } }
I20260619 06:39:29.254688 19803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } } }
I20260619 06:39:29.256994 19803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:29.256870 19676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:29.257058 19651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "234d24401a8340679bb59f6a95cd657e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "37eee7af3728462699f4e205f2830245"
I20260619 06:39:29.257558 19651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:29.261412 19651 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 234d24401a8340679bb59f6a95cd657e in term 1.
I20260619 06:39:29.262249 19650 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [term 1 FOLLOWER]: Refusing update from remote peer 234d24401a8340679bb59f6a95cd657e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:29.263235 19778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [LEADER]: Connected to new peer: Peer: permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:29.273885 19778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "234d24401a8340679bb59f6a95cd657e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } } }
I20260619 06:39:29.274199 19778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:29.241003 19535 master_runner.cc:186] Error getting master registration for 127.28.104.253:37279: OK, instance_id { permanent_uuid: "37eee7af3728462699f4e205f2830245" instance_seqno: 1781851168427226 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:39:29.281867 19535 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:39:29.289063 19784 catalog_manager.cc:1347] Generated new cluster ID: e201081bf69c4d64a99648f11cf40c3c
I20260619 06:39:29.289170 19784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:29.311147 19803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 234d24401a8340679bb59f6a95cd657e. Latest consensus state: current_term: 1 leader_uuid: "234d24401a8340679bb59f6a95cd657e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } } }
I20260619 06:39:29.311494 19803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:29.311851 19811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "234d24401a8340679bb59f6a95cd657e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } } }
I20260619 06:39:29.312211 19811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:39:29.324469 19773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "234d24401a8340679bb59f6a95cd657e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234d24401a8340679bb59f6a95cd657e" member_type: VOTER last_known_addr { host: "127.28.104.254" port: 41367 } } peers { permanent_uuid: "37eee7af3728462699f4e205f2830245" member_type: VOTER last_known_addr { host: "127.28.104.253" port: 37279 } } peers { permanent_uuid: "e745c2835f224f55b694b504d66a3d42" member_type: VOTER last_known_addr { host: "127.28.104.252" port: 43591 } } }
I20260619 06:39:29.324800 19773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:29.337769 19784 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:29.339746 19784 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:29.354952 19818 catalog_manager.cc:1259] Loaded cluster ID: e201081bf69c4d64a99648f11cf40c3c
I20260619 06:39:29.355129 19818 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245: loading cluster ID for follower catalog manager: success
I20260619 06:39:29.356244 19784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 234d24401a8340679bb59f6a95cd657e: Generated new TSK 0
I20260619 06:39:29.357298 19784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:29.359769 19818 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245: acquiring CA information for follower catalog manager: success
I20260619 06:39:29.362268 19818 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 37eee7af3728462699f4e205f2830245: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:39:29.490447 19757 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.470681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.28.104.254:41367)
0619 06:39:29.470884 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.470907 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.488228 (+ 17321us) server_negotiation.cc:407] Connection header received
0619 06:39:29.488731 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.488740 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.488830 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.488905 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.490108 (+ 1203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43160: BlockingRecv error: recv got EOF from 127.0.0.1:43160 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 06:39:29.492781 19754 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.472833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.28.104.252:43591)
0619 06:39:29.473002 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.473009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.491199 (+ 18190us) server_negotiation.cc:407] Connection header received
0619 06:39:29.491443 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.491449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.491531 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.491602 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.492525 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44624: BlockingRecv error: recv got EOF from 127.0.0.1:44624 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:39:29.496785 19680 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.494275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.28.104.253:37279)
0619 06:39:29.494471 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.494479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.495062 (+ 583us) server_negotiation.cc:407] Connection header received
0619 06:39:29.495308 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.495314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.495418 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.495487 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.496559 (+ 1072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50734: BlockingRecv error: recv got EOF from 127.0.0.1:50734 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:39:29.541821 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.533093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.28.104.253:37279)
0619 06:39:29.533383 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.533400 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.533905 (+ 505us) server_negotiation.cc:407] Connection header received
0619 06:39:29.534251 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.534257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.534474 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.534623 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:29.535380 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.536525 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.538612 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.539310 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.539407 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.540190 (+ 783us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:29.540943 (+ 753us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:29.541098 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.541354 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.541461 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260619 06:39:29.595634 19754 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.577833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.28.104.252:43591)
0619 06:39:29.578072 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.578083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.593721 (+ 15638us) server_negotiation.cc:407] Connection header received
0619 06:39:29.593969 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.593975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.594099 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.594193 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.595398 (+ 1205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44634: BlockingRecv error: recv got EOF from 127.0.0.1:44634 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:39:29.642105 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.627336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.28.104.253:37279)
0619 06:39:29.627579 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.627586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.634955 (+ 7369us) server_negotiation.cc:407] Connection header received
0619 06:39:29.635441 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.635446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.635550 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.635617 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:29.636356 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.637594 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.639732 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.640439 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.640540 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.641213 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:29.641378 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:29.641441 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.641689 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.641785 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 06:39:29.685736 19754 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.675788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.28.104.252:43591)
0619 06:39:29.676026 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.676035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.683821 (+ 7786us) server_negotiation.cc:407] Connection header received
0619 06:39:29.684079 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.684085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.684184 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.684290 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.685456 (+ 1166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44636: BlockingRecv error: recv got EOF from 127.0.0.1:44636 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:39:29.706274 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.699235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.28.104.253:37279)
0619 06:39:29.699427 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.699435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.700011 (+ 576us) server_negotiation.cc:407] Connection header received
0619 06:39:29.700272 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.700279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.700377 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.700481 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:29.700977 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.702113 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.704125 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.704789 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.704911 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.705476 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:29.705608 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:29.705696 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.705873 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.705952 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 06:39:29.715389 19754 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.712907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.28.104.252:43591)
0619 06:39:29.713114 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.713120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.713665 (+ 545us) server_negotiation.cc:407] Connection header received
0619 06:39:29.713991 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.713996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.714096 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.714171 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.715169 (+ 998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44640: BlockingRecv error: recv got EOF from 127.0.0.1:44640 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:39:29.730878 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.723539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.28.104.253:37279)
0619 06:39:29.723786 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.723794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.724314 (+ 520us) server_negotiation.cc:407] Connection header received
0619 06:39:29.724497 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.724506 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.724595 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.724690 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:29.725223 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.726427 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.728476 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.729309 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.729452 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.730053 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:29.730184 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:29.730284 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.730474 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.730559 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260619 06:39:29.740877 19754 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.738893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.28.104.252:43591)
0619 06:39:29.739146 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.739153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.739173 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:39:29.739461 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.739467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.739582 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.739694 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.740668 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44656: BlockingRecv error: recv got EOF from 127.0.0.1:44656 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
I20260619 06:39:29.761817 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.753693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.28.104.253:37279)
0619 06:39:29.753912 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.753919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.754542 (+ 623us) server_negotiation.cc:407] Connection header received
0619 06:39:29.754755 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.754764 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.754867 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.754972 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:29.755550 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.756727 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.759081 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.759974 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.760119 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.760865 (+ 746us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:29.761057 (+ 192us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:29.761167 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.761371 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.761464 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 06:39:29.772054 19754 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.770011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.28.104.252:43591)
0619 06:39:29.770449 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.770455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.770479 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:39:29.770555 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.770561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.770658 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.770786 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.771777 (+ 991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44660: BlockingRecv error: recv got EOF from 127.0.0.1:44660 (error 108)
Metrics: {"server-negotiator.queue_time_us":291}
I20260619 06:39:29.800626 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.792722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.28.104.253:37279)
0619 06:39:29.792939 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.792948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.793662 (+ 714us) server_negotiation.cc:407] Connection header received
0619 06:39:29.793938 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.793944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.794047 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.794177 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:29.794770 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.795973 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.798242 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.798935 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.799081 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.799688 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:29.799832 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:29.799951 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.800153 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.800241 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 06:39:29.809897 19754 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.807436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.28.104.252:43591)
0619 06:39:29.807632 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.807639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.808260 (+ 621us) server_negotiation.cc:407] Connection header received
0619 06:39:29.808453 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.808459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.808559 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.808674 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.809659 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44674: BlockingRecv error: recv got EOF from 127.0.0.1:44674 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:39:29.858496 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.847340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.28.104.253:37279)
0619 06:39:29.847586 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.847594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.852783 (+ 5189us) server_negotiation.cc:407] Connection header received
0619 06:39:29.853078 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.853086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.853213 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.853330 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:29.853858 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.854728 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.856343 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.856899 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.857097 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.857557 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:29.857702 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:29.857845 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.858017 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.858124 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 06:39:29.867213 19754 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.865373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.28.104.252:43591)
0619 06:39:29.865584 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.865590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.865836 (+ 246us) server_negotiation.cc:407] Connection header received
0619 06:39:29.866005 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.866011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.866111 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.866215 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.866984 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44676: BlockingRecv error: recv got EOF from 127.0.0.1:44676 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:39:29.941555 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:29.935569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.28.104.253:37279)
0619 06:39:29.935807 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.935814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.936214 (+ 400us) server_negotiation.cc:407] Connection header received
0619 06:39:29.936419 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.936424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.936492 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.936575 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:29.937082 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.937908 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.939819 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:29.940233 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:29.940379 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:29.940850 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:29.940952 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:29.941038 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:29.941200 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 06:39:29.941264 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:39:29.949180 19754 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:29.947135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.28.104.252:43591)
0619 06:39:29.947402 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:29.947408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:29.947749 (+ 341us) server_negotiation.cc:407] Connection header received
0619 06:39:29.947906 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:29.947910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:29.947975 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:29.948052 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:29.948895 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44686: BlockingRecv error: recv got EOF from 127.0.0.1:44686 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:39:30.049754 19770 catalog_manager.cc:1259] Loaded cluster ID: e201081bf69c4d64a99648f11cf40c3c
I20260619 06:39:30.049847 19770 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42: loading cluster ID for follower catalog manager: success
I20260619 06:39:30.052346 19770 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42: acquiring CA information for follower catalog manager: success
I20260619 06:39:30.054105 19770 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e745c2835f224f55b694b504d66a3d42: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:39:30.086606 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:30.080699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.28.104.253:37279)
0619 06:39:30.080937 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:30.080945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:30.081297 (+ 352us) server_negotiation.cc:407] Connection header received
0619 06:39:30.081555 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:30.081559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:30.081625 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:30.081727 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:30.082241 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:30.083001 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:30.084649 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:30.085196 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:30.085357 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:30.085787 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:30.085915 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:30.086028 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:30.086190 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 06:39:30.086276 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:39:30.097564 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:30.092222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.28.104.252:43591)
0619 06:39:30.092450 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:30.092456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:30.092702 (+ 246us) server_negotiation.cc:407] Connection header received
0619 06:39:30.092852 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:30.092856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:30.093008 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:30.093154 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:30.093496 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:30.094267 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:30.095494 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:30.095887 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:30.095978 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:30.096402 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:30.096940 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:30.097110 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:30.097218 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 06:39:30.097280 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 06:39:30.110657 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:30.104331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.28.104.253:37279)
0619 06:39:30.104540 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:30.104546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:30.104846 (+ 300us) server_negotiation.cc:407] Connection header received
0619 06:39:30.105057 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:30.105063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:30.105148 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:30.105243 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:30.105891 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:30.106992 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:30.108870 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:30.109384 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:30.109527 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:30.109953 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:30.110045 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:30.110146 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:30.110291 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 06:39:30.110367 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:39:30.112274 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:30.103493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.28.104.252:43591)
0619 06:39:30.103698 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:30.103719 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:30.105900 (+ 2181us) server_negotiation.cc:407] Connection header received
0619 06:39:30.106181 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:30.106187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:30.106261 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:30.106319 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:30.107086 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:30.107778 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:30.109935 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:30.110562 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:30.110643 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:30.111505 (+ 862us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:30.111612 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:30.111708 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:30.111878 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 06:39:30.111959 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:39:30.112704 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:30.103397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.28.104.254:41367)
0619 06:39:30.105977 (+ 2580us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:30.105981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:30.106015 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:39:30.106102 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:30.106105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:30.106258 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:30.106433 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:30.107013 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:30.108287 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:30.110410 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:30.111010 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:30.111135 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:30.111658 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:30.112091 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:30.112201 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:30.112314 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 06:39:30.112416 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2470,"thread_start_us":2374,"threads_started":1}
I20260619 06:39:30.116585 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 19535
I20260619 06:39:30.118076 19597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:30.228886 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 19535
I20260619 06:39:30.250149 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 19609
I20260619 06:39:30.251266 19671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:30.357401 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 19609
I20260619 06:39:30.377883 29091 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu pid 19684
I20260619 06:39:30.379058 19746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:30.486275 29091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2JAv5/build/asan/bin/kudu with pid 19684
2026-06-19T06:39:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4396 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9296 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (364930 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST